A demo is a vibe. An eval
is a contract.
Every LLM feature that failed in production passed a demo first. This is the method we use to build evaluation suites that gate releases: a golden set drawn from real questions, graders you can trust, and a regression loop that turns every bad answer into a permanent test.
The failure mode.
Large language model (LLM) features fail in a specific, predictable way. The demo is impressive, the first week is quiet, and then the long tail arrives: questions phrased differently, edge cases nobody rehearsed, a model update that shifts behaviour overnight. Without measurement, every one of those surprises reaches a user before it reaches you.
An evaluation suite inverts that. It is a set of real questions with known-correct answers, run against the system on every change, with hard numbers that decide whether a release ships. Teams that have one iterate quickly and sleep well. Teams that do not have one re-test by hand, ship on hope, and find out in support tickets. We covered why this matters in the pillar essay; this guide is the how. Read the pillar essay.
The four numbers.
Before building anything, agree on what a passing release means. We track four numbers, and only four, on every run.
The share of golden cases where the answer matches the expected result, judged by the grader assigned to that case.
The share of answers whose citations actually support the claim. An answer that is right for the wrong reason is a latent failure.
The system should refuse when the knowledge is missing. Track both false confidence and false refusal; both erode trust.
Quality that arrives after the user gives up does not count. Watch the tail, not the average.
Build the golden set from real questions.
A golden set is a list of question-and-correct-answer pairs that represents what your users actually ask. Do not invent questions in a workshop. Harvest them: support tickets, sales call notes, search logs, the questions your own team asks the system in its first week. Thirty to fifty cases is enough to start, provided each one is real.
Every case needs the same anatomy: the question as a user would phrase it, the expected answer or the facts it must contain, the source documents that answer should be grounded in, the grader that will judge it, and a tolerance note for what counts as acceptable variation. Write the expected answer with the source open, not from memory.
Choose graders you can trust.
A grader is whatever decides that an answer passed. Use the cheapest grader that can actually judge the case, and escalate only when the case demands it.
- ·Exact match for codes, numbers, and names
- ·Schema validation for structured output
- ·Citation-presence and link-resolves checks
- ·Fast, free, and never wrong about what they check
- ·Expected facts listed per case, checked one by one
- ·Catches partial answers and invented extras
- ·Human-writable, machine-checkable
- ·The workhorse for knowledge questions
- ·A second model scores against the rubric
- ·Needed for tone, reasoning, and open-ended answers
- ·Known biases: position, verbosity, self-preference
- ·Calibrate against 20 human-graded cases first
The judge deserves suspicion. Before trusting an LLM-as-judge, run it over twenty cases a human has already graded and measure the agreement. Use a different model family for judging than for answering, keep the judge prompt versioned like code, and spot-check a sample of its verdicts every month. A judge you never audit is just a second opinion you never questioned.
Wire it into CI and let it say no.
Run the full suite on every change that can shift behaviour: prompts, models, retrieval settings, chunking, tool definitions. Publish the four numbers as a report artifact, compare against the last accepted run, and fail the pipeline on regression. The suite earns its keep the first time it blocks a bad release at ten in the morning instead of ten at night.
Close the loop: every bad answer becomes a test.
A user flags a wrong answer. Log the exact question, the answer given, and the sources retrieved, straight from the trace.
Decide what failed: retrieval missed, grounding ignored, the model guessed, or the knowledge simply does not exist.
Write the golden case with the correct answer and source, and add it to the suite. The failure is now permanent memory.
The fix is proven by the suite going green, and that question can never silently regress again.
Sizing and cost.
The first version is a two-to-three day build for one engineer: harvest thirty to fifty questions, write the expected answers with sources open, wire deterministic and rubric graders, and add the CI job. If you use an LLM judge, add a day to calibrate it against human-graded cases. From there, upkeep is the weekly triage half-hour.
Running cost is small and worth paying on every release: a 40-case suite with a judge model costs a few rupees per run at current API prices, and a private judge model makes it effectively free. Against one production incident traced in front of a client, the suite pays for itself the first month. When regulators or enterprise buyers ask how you verify your AI, and under the DPDP Act's due-diligence duties for significant fiduciaries they will, the suite's history is the evidence. See the DPDP Act field notes.
An evaluation suite gating every release of an enterprise agent.
The enterprise AI agent we built and handed over ships behind exactly this loop: a golden set harvested from real workflows, rubric and judge graders, and a CI gate that has blocked regressions before users ever saw them. The client's own team runs it today.
→ Read the case studyFive ways eval suites die.
Questions, answered.
01What is an LLM evaluation suite?+
A set of real questions with known-correct answers, run automatically against your AI system on every change. Each case is judged by a grader, the results roll up into a small set of numbers, and the release ships only if those numbers hold. It does for AI behaviour what a test suite does for code.
02How many test cases do we need to start?+
Thirty to fifty real cases beat thousands of synthetic ones. Harvest them from support tickets, sales calls, and logs so they reflect what users actually ask. The suite then grows organically: every bad answer reported in production is encoded as a new case, so coverage concentrates exactly where the product hurts.
03What is a golden set?+
The curated list of question-and-expected-answer pairs at the heart of the suite, each with its source documents, an assigned grader, and a tolerance note. It is called golden because the answers are verified against sources when written, not recalled from memory, and the set is version-controlled like code.
04What is LLM-as-judge and can we trust it?+
LLM-as-judge means using a second model to score answers against a rubric, which is how evaluation scales past what humans can grade by hand. It is trustworthy only after calibration: run it over cases humans have already graded, measure agreement, use a different model family than the one answering, and spot-check verdicts monthly. Uncalibrated judges have known biases toward verbose and familiar-sounding answers.
05Should we buy an eval tool or build our own?+
Start with the loop, not the tool. The golden set, the graders, and the ship-or-block rule are the substance, and they fit in a repo with a CI job in a few days. Dedicated platforms earn their place once volume grows, and because the golden set is plain data you can adopt one later without starting over. What does not work is buying a dashboard and skipping the discipline.
06How do we evaluate RAG systems specifically?+
Split the measurement in two. Retrieval is checked by whether the right documents were fetched for each golden case, and generation by whether the answer matches the expected facts and cites those documents. When a case fails, the split tells you immediately whether to fix search or prompting, which are different repairs with different owners.
07How do we evaluate AI agents that take actions?+
Grade trajectories, not just final answers. Each golden case defines the expected outcome plus constraints on the path: which tools may be called, in what order, and what must never happen, like a destructive action without confirmation. Deterministic checks on tool calls plus a judge on the final state cover most agent behaviour, and refusal correctness matters double because an agent that guesses acts on its guess.
08How often should the suite run, and what does it cost?+
On every change that can shift behaviour, which in practice means every pull request touching prompts, models, retrieval, or tools, plus a scheduled run to catch provider-side model updates. A 40-case suite with an LLM judge costs a few rupees per run on hosted APIs, and near nothing with a private judge model. The weekly triage half-hour is the real cost, and it is the highest-leverage half-hour in the AI roadmap.
Ship when the suite says so.
Bring one workflow and twenty real questions. In one conversation we can usually sketch the golden set, pick the graders, and size the build for your stack.
Start a conversation →