Build four tiers: assertions, golden datasets, LLM-as-judge, and human review — then run the first two on every commit. Public benchmarks measure general capability. They cannot tell you whether yesterday’s prompt edit broke the one workflow your customers pay for. Only your own suite does that.
You changed a prompt, shipped it, and three days later support tells you the summaries got worse. Nobody can say when it started or which change caused it, because nothing was measuring.
That is the problem an eval suite solves. Not proving your model is good, but detecting the day it stops being good.
At a glance
- Benchmark scores measure general capability, never your specific workflow
- Cheap deterministic assertions catch most real regressions
- LLM-as-judge is useful, biased, and must itself be validated against humans
- An eval suite that takes 40 minutes will be skipped; keep the CI tier under 5
- Regression detection matters more than absolute scores
- Every production failure should become a test case that day
Why do public benchmarks fail as regression tests?
Because they measure a different thing, and their own authors keep saying so.
The QLoRA team fine-tuned more than 1,000 models across 8 instruction datasets, and among their published conclusions was that current chatbot benchmarks are not trustworthy for accurately evaluating chatbot performance (Dettmers et al., arXiv:2305.14314, May 2023).
The pattern repeats in retrieval. The MTEB authors benchmarked 33 models across 58 datasets and found no method dominated across all tasks (Muennighoff et al., arXiv:2210.07316). A single leaderboard number compresses away exactly the variation you care about.
There is also a mechanical problem. Benchmarks are static and public, so they leak into training data over time, and a rising score can reflect contamination rather than capability.
Figure 1 — What each kind of evaluation can actually tell you
Public benchmark
“Is this model broadly capable compared to other models?”
Useful once, when selecting a model. Static, public, contaminable, and averaged across tasks you do not run.
Selection tool
Your eval suite
“Did today’s change break the thing customers pay for?”
Private, task-specific, versioned with your code, and rerun on every commit.
Regression tool
These are not competing options. Use a benchmark to shortlist a model, then never look at it again. From that point on, only your own suite can tell you whether the system is getting better or worse.
What are the four tiers?
Each tier costs more and catches different failures. Run them at different frequencies.
Figure 2 — The four tiers, cheapest first
- TIER 1AssertionsDeterministic checks. Milliseconds. Every commit.
- TIER 2Golden datasetFixed inputs, known-good outputs. Minutes. Every commit.
- TIER 3LLM-as-judgeGraded quality on open-ended output. Slower, costs tokens. Nightly.
- TIER 4Human reviewExpensive, authoritative, and what validates tier 3. Weekly.
The ordering is a budget decision. Tiers 1 and 2 are cheap enough to gate every merge. Tiers 3 and 4 are not, so they run on a schedule and inform the cheaper tiers rather than blocking developers.
Tier 1 — assertions
Deterministic, instant, and far more valuable than their simplicity suggests.
- Output parses as valid JSON against the expected schema
- Required fields are present and non-empty
- Numeric values fall inside plausible bounds
- No placeholder text such as “lorem” or “TODO” survives into output
- Response length sits within an expected band
- Forbidden strings, including competitor names or internal identifiers, do not appear
A surprising share of production incidents are caught here, because most real failures are structural rather than subtle. The model returns prose where JSON was expected, or omits a field, and everything downstream breaks.
Tier 2 — the golden dataset
Fifty to two hundred fixed inputs with outputs you have verified are correct. This is the backbone.
For extraction and classification, compare exactly. For generation, compare on properties: does the summary mention the three facts it must mention, and stay under the length limit?
Build it from real production traffic, weighted toward the awkward cases. Clean inputs pass everything and teach you nothing.
Tier 3 — LLM-as-judge
Use a model to grade open-ended outputs no assertion can score. Coherence, tone, helpfulness, faithfulness to a source document.
The QLoRA authors found GPT-4 evaluations to be a cheap and reasonable alternative to human evaluation, which is the strongest available argument for the technique. It is also an argument with limits, covered below.
Tier 4 — human review
Expensive and irreplaceable. Its main job is not grading your system; it is grading your judge.
How do you keep LLM-as-judge honest?
By treating the judge as a measuring instrument that needs calibration, not as an oracle.
| Known bias | What it does | Mitigation |
|---|---|---|
| Position bias | Favours whichever response appears first in a pairwise comparison | Run both orderings and average |
| Length bias | Scores longer answers higher regardless of quality | Control for length; check score against word count |
| Self-preference | Rates output from its own model family more favourably | Judge with a different model family than the one generating |
| Scale compression | Clusters everything at 4 out of 5, hiding real differences | Use binary or three-point rubrics with explicit criteria |
| Criteria drift | Applies vague instructions inconsistently across runs | Give concrete pass and fail examples inside the judge prompt |
Validate the judge before you trust it: have humans grade 50 outputs, have the judge grade the same 50, and measure agreement. Below roughly 80% agreement the judge is measuring something other than what you intended, and its scores should not gate anything.
Watch for this
Never use the same model as both generator and judge in the same family without checking self-preference. A model asked to grade its own output tends to like it. This is the single most common way an eval suite ends up reporting improvement while quality falls.
What should actually gate a deploy?
Fewer things than teams initially specify, or the suite becomes an obstacle people route around.
| Tier | Runs | Blocks merge? | Typical budget |
|---|---|---|---|
| Assertions | Every commit | Yes, always | Under 30 seconds |
| Golden dataset | Every commit | Yes, on regression | 2 to 5 minutes |
| LLM-as-judge | Nightly and pre-release | No, alerts only | Token cost per run |
| Human review | Weekly sample | No | 1 to 2 hours of expert time |
Define regression as a drop against the previous run, not against an absolute threshold. Absolute thresholds get argued down over time. A relative drop is a fact.
How do you handle non-determinism?
The complaint that stops most teams building evals at all: the same input produces different output each run, so how can anything be a test?
Three answers, in order of usefulness.
- Set temperature to zero for evals. Not fully deterministic in practice, but it removes most variance.
- Test properties, not strings. “Contains the refund amount” is stable where exact-match is not.
- Run n times and threshold. For genuinely stochastic behaviour, run five times and require four passes. Flakiness becomes a measured rate rather than a mystery.
Track the flake rate itself. A test that starts flaking is often reporting a real degradation before your metrics do.
Where do eval suites go wrong?
Measuring what is easy instead of what matters
Exact-match scoring on a generation task is convenient and close to meaningless. It punishes correct answers phrased differently and rewards memorised phrasing.
If a metric cannot distinguish a good answer from a bad one when you inspect the cases manually, it is decoration.
Letting the suite get slow
An eval suite taking 40 minutes will be skipped under deadline pressure, and a skipped suite has zero value. Keep the blocking tier under five minutes even if that means sampling.
Never updating it
A golden dataset frozen at launch decays as the product changes. The discipline that keeps it alive is simple: every production failure becomes a test case the same day it is diagnosed.
Teams that do this end up with a suite shaped like their actual failure modes rather than their initial assumptions.
What do experienced teams do differently?
They version the eval suite alongside the prompt, in the same commit.
A prompt change and its corresponding eval change ship together, so the history shows what was expected to change and what was not. Reviewing a prompt diff without the eval diff is reviewing half the change.
They also keep a small adversarial set separate from the golden dataset: prompt injections, contradictory instructions, empty inputs, enormous inputs, wrong-language inputs. It rarely improves, which is the point. It exists to detect the day something regresses.
A short glossary
- Golden dataset
- A fixed set of inputs with verified correct outputs, used as the stable reference for regression detection.
- LLM-as-judge
- Using a language model to grade another model’s output against a rubric, standing in for human review at lower cost.
- Position bias
- A judge’s tendency to favour whichever candidate response it sees first in a pairwise comparison.
- Benchmark contamination
- Leakage of public benchmark data into training corpora, inflating scores without improving real capability.
- Flake rate
- The proportion of runs where a non-deterministic test fails despite no underlying change.
Three things to do next
- Write ten assertions this week. Schema validity, required fields, length bounds. It takes an afternoon and catches structural failures immediately.
- Collect fifty real inputs from production logs and label the correct output for each. Weight toward cases that already caused complaints.
- Wire tiers 1 and 2 into CI with a hard five-minute budget. Add the judge tier only once the cheap tiers are green and trusted.
Key takeaways
- Public benchmarks select models; only a private suite detects regressions in your workflow.
- The QLoRA authors concluded current chatbot benchmarks are not trustworthy for accurately evaluating chatbot performance.
- Four tiers — assertions, golden dataset, LLM-as-judge, human review — cost and catch different things.
- Deterministic assertions catch a large share of real incidents because most failures are structural.
- Validate any judge against human grades on 50 outputs and expect at least 80% agreement before trusting it.
- Never judge with the same model family that generated the output without checking self-preference.
- Keep the blocking CI tier under five minutes, or it will be skipped when it matters most.
Frequently asked questions
How many examples does an eval suite need?
Fifty to two hundred for the golden dataset is enough to detect meaningful regressions. Quality matters far more than volume: examples drawn from real failures are worth many times more than synthetic ones. Start with fifty real cases and grow the set from production incidents.
Can I use the same model to generate and judge?
Only after testing for self-preference, since models tend to rate output from their own family more favourably. Where possible, judge with a different model family than the one generating. If you cannot, validate judge scores against human grades before letting them gate anything.
How do I evaluate outputs that have no single correct answer?
Score properties rather than exact text. Check whether required facts appear, whether length constraints hold, and whether the output stays faithful to its source. For subjective qualities like tone, use an LLM judge with a binary or three-point rubric containing concrete pass and fail examples.
What is benchmark contamination?
Leakage of public benchmark questions into model training data, which inflates scores without any real capability gain. It is a structural weakness of static public benchmarks and one reason a private eval suite built from your own traffic is more trustworthy for tracking change over time.
Should eval failures block deployment?
Assertions and golden dataset regressions should block. LLM-as-judge scores should alert rather than block, because judge variance can produce false failures that erode trust in the whole suite. Human review runs on a sample and informs direction rather than gating individual releases.
How do I deal with non-deterministic output in tests?
Set temperature to zero for eval runs, test properties instead of exact strings, and for genuinely stochastic behaviour run the case several times and require a pass threshold such as four out of five. Track the flake rate, since a rising rate often signals real degradation.
How often should the golden dataset be updated?
Continuously. Add a test case every time a production failure is diagnosed, so the suite grows to match your actual failure modes rather than your launch-day assumptions. A dataset frozen at launch steadily loses relevance as the product changes around it.
Is LLM-as-judge accurate enough to rely on?
It is useful and measurably biased. The QLoRA authors described GPT-4 evaluation as a cheap and reasonable alternative to human evaluation, but known biases around position, length and self-preference mean a judge must be calibrated against human grades and re-checked whenever the judge model changes.
References
- Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314, submitted 23 May 2023. Source for the findings on GPT-4 as evaluator and on chatbot benchmark trustworthiness.
- Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. MTEB: Massive Text Embedding Benchmark. arXiv:2210.07316, submitted 13 October 2022, revised 19 March 2023.
- Liang, P., et al. Holistic Evaluation of Language Models (HELM). arXiv:2211.09110.
