Treat an LLM judge as an instrument requiring calibration, not an oracle delivering truth. Validate it against human labels on at least 50 outputs, expect 80% agreement before trusting it, and re-validate whenever the judge model changes. Uncalibrated judges reliably report improvement while quality falls.
The appeal is obvious. Human grading is slow and expensive, and a model can score a thousand outputs overnight for the price of a coffee.
The QLoRA authors found GPT-4 evaluations to be a cheap and reasonable alternative to human evaluation (arXiv:2305.14314). In the same paper they also concluded that current chatbot benchmarks are not trustworthy for accurately evaluating chatbot performance. Both statements are true, and holding them together is the whole discipline.
What is an LLM judge good at?
Relative comparison, not absolute scoring.
Asked whether response A or response B better answers a question, a strong model agrees with human preference at a useful rate. Asked to rate a single response out of ten, it produces a number with poor calibration and little stability across runs.
| Task | Judge reliability | Why |
|---|---|---|
| Pairwise preference | Good | Relative comparison needs no absolute scale |
| Binary pass or fail against a rubric | Good | Two options with explicit criteria |
| Faithfulness to a source document | Good | Verifiable against supplied text |
| Three-point rubric | Acceptable | Coarse enough to stay stable |
| Score out of 10 | Poor | Clusters at 7 and 8, unstable between runs |
| Factual accuracy without a source | Poor | Judge inherits the same knowledge gaps |
The last row is the one that causes real damage. A judge cannot verify a claim it does not know, so it grades confident-sounding wrong answers as correct. Judging factuality requires supplying the ground truth in the prompt.
Which biases are documented and reproducible?
Figure 1 — The same comparison, two orderings
Single-order evaluation
Response A shown first, B second. Judge prefers A.
You record a win for A and move on, having measured position as much as quality.
Confounded
Both orderings, averaged
Run once as A then B, again as B then A. If the winner flips, the judge is expressing position bias, not preference.
Disagreement between orderings is itself a signal: it marks the pairs that are genuinely close.
Measurable
Cost note: this doubles your judge calls, which is why teams skip it. It is the single highest-value correction available, and skipping it silently corrupts every pairwise result you produce.
| Bias | Effect | Correction |
|---|---|---|
| Position | Favours the first response shown | Run both orderings, average, flag disagreements |
| Length | Rates longer answers higher regardless of content | Correlate score against word count; if correlation is strong, the rubric is measuring length |
| Self-preference | Prefers output from its own model family | Judge with a different family than the generator |
| Scale compression | Clusters ratings at 4 of 5, hiding differences | Use binary or three-point scales |
| Authority cues | Rewards confident phrasing over correctness | Require the judge to cite the specific span supporting its verdict |
| Formatting | Prefers bulleted or structured output | Normalise formatting before judging where format is not the thing being scored |
How do you calibrate a judge?
- Sample 50 to 100 outputs spanning the quality range, including known-bad ones. A sample of only good outputs cannot reveal whether the judge discriminates.
- Have humans label them against the same rubric you will give the judge. Two labellers on an overlapping subset gives you human-to-human agreement as your ceiling.
- Run the judge on the identical set, blind to the human labels.
- Measure agreement and inspect every disagreement individually. The disagreements are where the rubric is ambiguous.
- Revise the rubric, not the judge model, then repeat. Most calibration failures are specification failures.
Compare judge-to-human agreement against human-to-human agreement rather than against 100%. If your own labellers agree only 85% of the time, a judge hitting 82% is performing close to the ceiling and the task is inherently subjective.
Watch for this
A judge that agrees with humans 95% of the time on a dataset where 95% of outputs are good has learned to say “pass” and nothing more. Always check agreement separately on the failing subset. Overall accuracy on an imbalanced set is close to meaningless.
What makes a rubric that actually discriminates?
Concrete criteria and worked examples. Vague adjectives produce vague grading.
A prompt asking the judge to rate “helpfulness” invites it to apply its own shifting definition. A prompt stating that a response fails if it omits any of three required elements, with one passing and one failing example shown, produces stable and inspectable judgments.
- State the criteria as checkable conditions, not qualities
- Include one clear pass example and one clear fail example
- Require the judge to quote the span justifying its verdict
- Ask for the reasoning before the verdict, since generation runs left to right
- Keep the scale coarse: binary where possible, three points at most
The reasoning-before-verdict ordering matters more than it sounds. Asking for a score first forces the judge to commit before analysing, and the explanation that follows becomes a rationalisation of an already-fixed answer.
How do you detect judge drift?
Judge scores can move for three reasons, and only one of them is your system.
Figure 2 — Interpreting a score change
- Your system
The signal you wanted.Confirm the anchor set stayed flat - The judge model
Provider updated it under you.Re-run calibration before believing anything - The rubric
Someone edited the prompt.Version rubrics like code
The mechanism that separates these: keep an anchor set of roughly 20 outputs with fixed human labels and re-score them on every judge run. If anchor scores move while the outputs did not, your instrument drifted, not your product.
Pin the judge model version explicitly. A silent provider upgrade can shift scores across your entire history, and without an anchor set you will spend days investigating a product regression that never happened.
Where do teams go wrong?
Letting judge scores gate deployment
Judge variance produces false failures. A blocked deploy caused by measurement noise erodes trust in the whole eval system, and once developers start overriding it routinely, the suite has no authority left. Alert on judge scores; block on deterministic checks.
Judging with the generating model
Self-preference is well documented and easy to avoid. Where a single provider is unavoidable, at minimum test the magnitude of the effect by scoring a known-mixed set and checking whether same-family output wins more than it should.
Optimising against the judge
Once a judge scores your releases, people start tuning for it. Because judges reward length and structure, systems drift toward verbose, heavily formatted output that scores well and serves users worse. Periodic human review is the only reliable check on this.
What do experienced teams do differently?
They keep a permanent human sample, however small.
Twenty outputs reviewed by a person every week costs an hour and anchors everything else. Without it, an eval system can drift for months in a direction that looks like improvement on every dashboard.
They also version the rubric alongside the code that uses it and store the judge’s reasoning, not just its verdict. When a score is disputed later, the reasoning usually reveals whether the judge misread the output or the rubric was ambiguous, and those need different fixes.
A short glossary
- Pairwise preference
- Asking a judge which of two responses is better, rather than assigning either an absolute score.
- Position bias
- A judge’s tendency to favour whichever response appears first in a comparison.
- Self-preference bias
- A judge rating output from its own model family more favourably than output from others.
- Anchor set
- A small fixed set of outputs with stable human labels, re-scored each run to detect judge drift.
- Scale compression
- The clustering of ratings into a narrow band, hiding real quality differences between outputs.
- Inter-annotator agreement
- How often independent human labellers assign the same label, setting the realistic ceiling for judge accuracy.
Key takeaways
- An LLM judge is a measuring instrument with reproducible biases, not a source of ground truth.
- Judges are reliable at pairwise comparison and binary rubrics, unreliable at absolute scores out of ten.
- Position bias is corrected by running both orderings and averaging, which doubles cost and is worth it.
- Never judge factual accuracy without supplying the ground truth, since the judge shares the same knowledge gaps.
- Compare judge-to-human agreement against human-to-human agreement, not against perfection.
- Keep an anchor set of fixed-label outputs to separate judge drift from real product change.
- Alert on judge scores and block on deterministic checks, or measurement noise will destroy trust in the suite.
Frequently asked questions
How accurate is LLM-as-judge compared with human evaluation?
On pairwise comparison with a clear rubric, agreement with humans is often close to the agreement humans reach with each other. On absolute scoring it is considerably weaker. The QLoRA authors described GPT-4 evaluation as a cheap and reasonable alternative to human evaluation, which is endorsement with limits rather than equivalence.
Should I use a scale of 1 to 10?
No. Ten-point scales compress in practice, with most outputs landing on 7 or 8 and little stability between runs. Binary pass or fail against explicit criteria discriminates far better, and a three-point scale is the most granularity worth attempting for subjective qualities.
Can a model judge its own output?
It can, but self-preference bias means it will tend to rate its own family’s output more favourably. Use a different model family for judging where possible. If constrained to one provider, measure the size of the effect on a mixed set before trusting any comparative result.
How do I stop position bias affecting results?
Run every pairwise comparison twice with the order reversed, then average. When the winner flips between orderings, the judge is expressing position preference rather than quality preference, and those pairs should be treated as ties or escalated to human review.
How many human labels do I need to calibrate a judge?
Fifty to a hundred spanning the full quality range, deliberately including known-bad outputs. A sample of only good outputs cannot show whether the judge discriminates. Check agreement separately on the failing subset, since overall accuracy on an imbalanced set is misleading.
Can LLM judges verify factual accuracy?
Only against a supplied source. A judge asked to assess factuality from its own knowledge inherits the same gaps and confidently passes plausible-sounding errors. Provide the reference document and ask the judge to check consistency with it, which turns an unreliable task into a reliable one.
Should judge scores block a deployment?
No. Judge variance produces occasional false failures, and blocked deploys caused by measurement noise quickly erode trust in the entire evaluation system. Use judge scores as alerts and trend signals, and reserve hard gates for deterministic assertions and golden dataset regressions.
What is judge drift and how do I detect it?
Judge drift is a change in scores caused by the judge model or rubric changing rather than your system. Detect it with an anchor set of roughly 20 outputs carrying fixed human labels, re-scored on every run. If anchor scores move while those outputs did not, the instrument drifted.
References
- Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314, 2023. Source for both the endorsement of GPT-4 as evaluator and the caution on benchmark trustworthiness.
- Zheng, L., et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685. Documents position, verbosity and self-enhancement bias.
- Liang, P., et al. Holistic Evaluation of Language Models (HELM). arXiv:2211.09110.
