LLM-as-judge calibration checklist
The core problem with LLM judges is not that they are inaccurate - it is that they are asymmetrically inaccurate in the direction that flatters you. Jain, Ahmed, Sahai, and Leong's agreeableness-bias paper ('Beyond Consensus', arXiv:2510.11822) measured judges at a true positive rate of 96% on valid outputs but a true negative rate below 25% on invalid ones: the judge waves through most of what is wrong while confirming what is right, so an unvalidated judge systematically overstates your quality. The foundational MT-Bench study (Zheng et al., arXiv:2306.05685) set the realistic ceiling - GPT-4 agreed with human preferences about 85% of the time, while humans agreed with each other about 81% - and documented position and verbosity bias along the way; Ye et al.'s CALM framework later catalogued twelve distinct judge bias types. Even your validation protocol can mislead you: a NeurIPS 2025 paper on rating indeterminacy ('Validating LLM-as-a-Judge Systems under Rating Indeterminacy') showed across 11 rating tasks and 9 commercial LLMs that standard forced-choice validation can select judges up to 31% worse than validation that lets raters mark multiple acceptable answers. This checklist is the gate list between 'we added an LLM judge' and 'we trust its scores'.
AUDIENCE
Engineers and eval leads putting an LLM judge into a pipeline where its scores gate deploys, rank models, or feed dashboards executives act on
OUTCOME
A judge whose agreement with human experts is measured - not assumed - whose known biases are audited, and whose scores are trusted only within the limits that measurement supports
SECTION 01
4 CHECKS
Judge Design Before Any Calibration
Use binary pass/fail per criterion, not a 1-5 Likert scale
Hamel Husain's evals FAQ (hamel.dev) is blunt on this: the expert's question is 'did the AI achieve the desired outcome', and binary labels are clearer to calibrate and act on than subjective scales. If you need granularity, add more binary checks for sub-criteria - do not average a 3.7 out of an ill-defined 5. A Likert judge also hides the agreeableness failure mode: a score of 4 on a wrong answer is invisible; a 'pass' on a wrong answer is a countable error.
Write the rubric from real failures, not from imagination
The critique-shadowing workflow (hamel.dev, 'Creating a LLM-as-a-Judge That Drives Business Results') starts by pulling actual production outputs and having a principal domain expert judge them with written critiques. Rubrics invented before looking at real outputs encode the failures you expected, not the ones you have - and the judge inherits the blind spot.
Prefer deterministic checks over a judge wherever one exists
If the property is checkable by code - JSON validity, schema conformance, required citation present, exact-match answer, tests pass - a deterministic check wins over any LLM judge: zero variance, zero cost, zero bias. Guides like Latitude's CI/CD evaluation guide explicitly order the pipeline deterministic-first, judge-second. Reserve the judge for genuinely fuzzy criteria (tone, faithfulness, helpfulness) and keep the deterministic layer as the hard gate.
Pin the judge configuration: model version, prompt, temperature 0
A judge whose model silently upgrades is a new judge with your old validation certificate. Pin the model version, version-control the judge prompt, and run scoring at temperature 0 for reproducibility (Latitude's guide makes the temperature-0 recommendation explicit). Any change to model, prompt, or rubric invalidates prior calibration and re-triggers this checklist.
SECTION 02
4 CHECKS
Bias Audits
Audit for agreeableness: measure TNR on known-bad outputs, not just overall agreement
The arXiv:2510.11822 numbers - TPR 96%, TNR <25% - mean a judge can post high overall agreement while missing three quarters of invalid outputs, because most production outputs are valid and the aggregate hides the asymmetry. Build a deliberate set of known-bad cases (real failures plus synthetically corrupted answers) and report the judge's true negative rate separately. A judge that cannot say 'fail' is a rubber stamp with an API bill.
Run position-swap tests on any pairwise judge
MT-Bench (Zheng et al.) documented position bias directly: judges prefer answers by position, not just content. Score every pair twice with the order swapped and count verdict flips; flipped verdicts are noise, not signal. Later systematic reviews add a subtle trap - high run-to-run stability can mask consistent position bias, so stability alone is not evidence of fairness.
Check verbosity and self-preference bias against your rubric
MT-Bench also documented verbosity bias - longer answers win without being better - and Ye et al.'s CALM framework catalogues twelve bias types including self-preference (judges favoring outputs from their own model family). If your judge model shares a family with a model being judged, either switch judge families or measure the self-preference delta explicitly before comparing models with it.
Validate with response-set ratings, not only forced choice, when multiple answers are acceptable
The NeurIPS 2025 rating-indeterminacy paper showed that forcing human raters to pick exactly one 'correct' label - when several responses are genuinely acceptable - biases which judge looks best, selecting judges up to 31% worse across its 11-task, 9-LLM study. If your task has indeterminate ratings (most generation tasks do), let validation raters mark all acceptable options and score the judge against the set.
SECTION 03
4 CHECKS
Human Calibration Protocol
Have one principal domain expert label a representative set before the judge scores anything
Critique shadowing (hamel.dev) names a single principal expert - not a committee - who makes pass/fail judgments with written critiques on real outputs. Start with a few dozen representative examples (practitioner summaries of the workflow commonly begin around 30 and expand), covering both typical traffic and known failure modes. The labeled set is the ground truth every judge iteration is measured against; without it, 'the judge seems right' is vibes.
Iterate judge prompt against expert labels until agreement is high - then keep a held-out set
Run the judge over the labeled set, do error analysis on disagreements, fix the rubric or prompt, repeat - that loop is the entire method. Practitioner guides summarizing the workflow target roughly 90%+ agreement with the expert before deployment; for context, MT-Bench's human-human agreement was about 81%, so demanding much more than the human ceiling means you are overfitting to one expert's quirks. Hold out labeled examples the prompt was never tuned on for the final number.
Report chance-corrected agreement (Cohen's kappa), not just raw percent
On imbalanced data - and production outputs are heavily pass-skewed - raw agreement is inflated by the base rate: a judge that always says 'pass' agrees 90% of the time when 90% of outputs pass. Systematic reviews of judge evaluation recommend Cohen's kappa or Krippendorff's alpha precisely because they correct for chance agreement. Report both, gate on kappa.
Schedule recalibration: judges drift when traffic, prompts, or models change
Calibration is a certificate about a distribution, not a property of the judge. New user segments, a new app prompt, or a judge-model update all shift the distribution the certificate covered. Re-run a labeled sample on a schedule (monthly, or per major release) and compare agreement against the original baseline - silent drift in a deploy-gating judge is a slow-motion incident.
SECTION 04
3 CHECKS
Production Guardrails
Gate deploys on deterministic checks plus judge trends - not on a single judge score
Given a sub-25% measured TNR on invalid outputs for uncalibrated judges (arXiv:2510.11822) and run-to-run variance inherent to sampling, a single judge score is too noisy to block or approve a release alone. Hard-block on deterministic checks; treat judge metrics as trend signals with human review on regression. If a judge score must gate, gate on aggregates over many cases with a baseline-relative threshold.
Never attach money or performance consequences to judge scores without a standing human audit
The moment a judge score pays a bonus, ranks a vendor, or scores an employee, every documented bias becomes an exploit surface - verbosity bias alone means 'write longer' beats 'write better'. Goodhart's law applies with published attack vectors. Keep a recurring human audit of a random sample of judged cases, and disclose the judge's measured agreement rate alongside any consequential score.
Log judge critiques and sample disagreements weekly
Make the judge output a written critique with its verdict, and route a sample of low-confidence or borderline cases to a human queue. The critiques are your error-analysis raw material and your early-warning system: when critique quality degrades or disagreement rates climb, calibration has slipped before the dashboard shows it.
Common mistakes
- 01Trusting overall agreement while never measuring the true negative rate - the exact blind spot the agreeableness-bias paper quantified at TPR 96% vs TNR <25%.
- 02Using a 1-5 Likert judge and averaging the scores, which hides both bias and disagreement inside a number nobody can act on.
- 03Validating with forced-choice labels on tasks where several answers are acceptable - the NeurIPS 2025 result showing this can select judges up to 31% worse.
- 04Shipping a judge with zero human-labeled ground truth because 'GPT-4 agreement with humans is 85% in MT-Bench' - that number is for MT-Bench's tasks, not yours.
- 05Letting an LLM judge do a job a regex, schema validator, or test suite could do deterministically for free.
- 06Silently upgrading the judge model mid-quarter and comparing scores across the boundary as if they were the same instrument.
- 07Reporting raw percent agreement on pass-skewed data instead of chance-corrected kappa, and mistaking base-rate inflation for judge quality.
- 08Wiring judge scores directly into deploy blocks or compensation before running a single bias audit.