Most teams evaluate prompts in a notebook. This is what belongs in the suite that gates releases instead: the dataset, the checks, the judge, the review and the gate.
Most first evaluation setups look the same: a spreadsheet of example prompts, a reviewer's gut feel, maybe a score from a model asked "how good was this answer?". It holds up until the second or third prompt change, at which point nobody can say whether quality moved.
A suite that can gate a release looks different, and its parts are recognizable. Here they are.
The dataset has to come from what users actually do, including the awkward parts: multi-part requests, odd phrasings, questions the system should decline or redirect. A dataset of happy-path examples passes everything and proves nothing.
In practice it grows from three sources: sampled real transcripts (anonymized where required), failure reports and support tickets, and adversarial cases written after a first pass at the failure taxonomy.
A surprising share of "AI quality" needs no model in the loop to check: was the tool call valid, does the output match the schema, are required citations present, is banned content absent, did the response stay inside latency and token budgets. These checks are cheap and stable, and they belong in CI next to the unit tests.
LLM-as-judge is useful for the fuzzy middle: tone, completeness, reasoning quality. It is dangerous when its scores are treated as ground truth. Before it gates anything, calibrate it: label a sample of cases by hand, measure agreement against those labels, then write down where the judge is wrong and either fix the rubric or shrink what it is allowed to judge.
Recalibrate whenever the judge model changes. A judge that was never validated is an opinion with an API key.
High-stakes categories get human-labeled gold sets and periodic audits. The point is not to review everything; it is to keep the automated layers honest.
A suite that produces a monthly PDF is archaeology. It should run on every change that can affect behavior (prompt, model, retrieval config, temperature) and block release when agreed thresholds fail. That is the difference between evaluation as documentation and evaluation as infrastructure.
"Bad output" is not a category. Unsupported claim, missed constraint, wrong tool, truncated context, formatting regression: named failure modes can be counted, prioritized, and re-tested after a fix ships.
Model and vendor comparisons stop being gambles, because you can compare candidates on your own tasks with your own thresholds. Prompt changes become changes with tests. Quality arguments end at the dashboard instead of with whoever argued last.
Suites have maintenance costs, and pretending otherwise is how they die: someone curates the dataset (a few hours a week at steady state), thresholds need a first round of judgment calls, and review time is real.
The usual killer is dataset decay. Add new failure cases the week you find them, prune stale ones every quarter, and treat the dataset as a fixture of the codebase rather than a document. Do that and the suite stays small enough to maintain and sharp enough to trust.