LLM-as-a-Judge for Production Evals: Building a Judge You Can Trust
Build an LLM-as-a-judge you can trust in production: calibrate against a golden set, control the biases, and know when it should not be the gate.
An LLM judge that agrees with itself is not the same as one that agrees with your users. GPT-4-class judges reach over 80% agreement with human raters on the benchmark that made the technique popular, but that number only holds after you calibrate against a labeled set and control for the biases that quietly inflate every score.
Use an LLM-as-a-judge to evaluate open-ended output at a scale humans cannot reach: chatbot replies, RAG answers, agent traces, summaries. Treat its score as a calibrated proxy for human judgment, never as ground truth. The strength of the judge model buys you a good starting point; the trust comes from a labeled golden set, a scale that reproduces, and explicit control of the biases that push scores in one direction. Skip that work and you get a number that looks like a metric and behaves like noise.
This is the build guide: pick the scale, calibrate against humans, control the biases, audit judge stability, and know the cases where a judge should never be the gate.
The one thing that separates a judge you can trust
A judge is a classifier you wrote in a prompt, and like any classifier it has an error rate. On the MT-Bench study that popularized the method, a strong judge (GPT-4) matched crowdsourced human preferences over 80% of the time, the same level of agreement humans reach with each other (Zheng et al., 2023). That is the ceiling, and it is only reachable on pairwise preference tasks with a calibrated setup.
The failure most teams make is to write a judge prompt, watch it produce plausible scores, and ship it as a metric. It will produce a number for every input. Whether that number tracks reality is a separate question, and the only way to answer it is to compare the judge against a human-labeled set. Everything below is in service of that comparison.
Pick a scale that actually reproduces
Use a binary or low-cardinality rubric, not a 1-10 score. Binary evaluations ("Concise" vs "Verbose", "Grounded" vs "Unsupported") are measurably more reliable and consistent for both LLMs and human raters than fine-grained numeric scales (EvidentlyAI). Ask two people, or two model calls, to decide whether a reply scores 73 or 82 on "helpfulness" and you will not get the same answer twice. Ask whether it fully answered the question, yes or no, and you will.
When you need gradation, use a defined 1-5 Likert scale where every level has an explicit, written definition. The moment a rater cannot say what separates a 3 from a 4, the scale stops reproducing and the judge starts leaning toward whichever score was common in its training data.
Here is the same criterion written two ways. The first does not reproduce; the second does.
# Weak: unbounded, undefined
Rate the helpfulness of this support reply from 1 to 10.
# Strong: bounded, each level defined, one axis
Score the reply on RESOLUTION using exactly one label:
- RESOLVED : fully answers the user's question, no missing step
- PARTIAL : correct but leaves a required step or caveat unstated
- UNRESOLVED : wrong, off-topic, or punts without answering
Judge only resolution. Ignore tone and length.
Return: {"label": "...", "reason": "<one sentence>"}Two rules are doing the work here. One axis per judge: a prompt that grades helpfulness, tone, and safety at once produces a blended score you cannot act on, so run three narrow judges instead of one wide one. And force a short reason before the label, which both gives you an audit trail and measurably improves the verdict, the core finding behind G-Eval's chain-of-thought judging.
Calibrate against a golden set, or you are guessing
Build a human-labeled golden set before you trust a single automated score. This is the step teams skip and the reason their eval dashboards lie. The golden set is a few hundred real inputs, spanning your actual failure modes, each labeled by a human using the exact rubric the judge will use. You are not evaluating the agent yet; you are evaluating the judge.
Sample real traffic, not happy-path demos
Pull 150 to 300 examples from production logs (or staging traffic that mirrors it). Oversample the edges: the long conversations, the ambiguous questions, the ones where the agent already failed. A judge calibrated only on easy cases falls apart exactly where you need it.
Label by hand with the rubric
Have a human apply the same labels the judge will output. Two labelers on an overlapping slice gives you human-human agreement, which is your realistic ceiling: if humans only agree 85% of the time, do not expect the judge to beat that.
Run the judge over the same set
Score every labeled example with your judge prompt. Keep the reasons; you will read the disagreements.
Compute agreement, then read the misses
Measure agreement between judge and human labels (raw agreement to start, Cohen's kappa once you want to account for chance). Then read every disagreement by hand. The patterns you see (the judge rewards length, or misses missing citations) are your next prompt revision.
Iterate the prompt, re-run, freeze
Revise the rubric, re-run, re-measure. When agreement clears your bar and holds, freeze that prompt and version it. That frozen prompt plus its measured agreement rate is your judge, and the agreement rate is the honest confidence interval on every score it will ever produce.
Once calibrated, log the judge's version, the score, and the reason alongside every trace so a score is always traceable back to the input and the rubric that produced it. That logging discipline is the same one that makes agent behavior debuggable in general, which we cover in what to log before an agent hits production.
Control the biases that inflate every score
Calibration surfaces the biases; now you suppress them. Three are well documented and show up in production constantly (Zheng et al., 2023):
- Position bias. In pairwise comparisons the judge favors whichever answer came first. Mitigate by running each comparison both ways (A vs B and B vs A) and only counting a win when it holds in both orders. The pairs that flip are your real ties.
- Verbosity bias. Longer answers get higher scores regardless of quality. Add an explicit instruction to ignore length, and when it persists, cap or normalize for token count in your rubric.
- Self-preference bias. A judge favors text from its own model family. G-Eval's authors flagged this directly: LLM evaluators lean toward LLM-generated output (Liu et al., 2023). When you are scoring output from model X, judge with a model from a different family.
Chain-of-thought and few-shot prompting help alignment but do not remove these biases. G-Eval, which pairs GPT-4 with explicit reasoning steps, reached a Spearman correlation of 0.514 with human scores on summarization, a large jump over older reference-based metrics and still far from perfect (Liu et al., 2023). Read that as: reasoning makes a judge better, calibration tells you how much better, and neither makes the judge into an oracle.
Audit judge stability, because the judge is a variable
Pin the judge model and version, and re-baseline whenever you change it. This is the part most eval setups miss. Recent 2026 work makes the point sharply: change the judge model and your measured performance changes with it, even when nothing about the agent moved. Your "the agent got 6% better" can be an artifact of the provider silently updating the model under your judge.
Treat the judge like production infrastructure. Pin an exact model version rather than a floating alias. Store that version with every score. When you must upgrade the judge, re-run it against the frozen golden set, confirm agreement still clears the bar, and re-baseline your historical comparisons on the new judge before you trust a trend across the switch. A judge you cannot reproduce is not a measurement instrument.
Where a judge should never be the gate
An LLM judge is for triage, monitoring, and regression testing at scale. It is not for decisions that carry real-world liability. Do not let a judge be the sole automated gate on:
- Safety- and compliance-critical output (medical, legal, financial advice): route these to human approval, and use the judge only to prioritize what the human reviews first.
- Exact factual or numeric correctness where a deterministic check exists: verify against the source or a unit test, not a probabilistic grader.
- Anything a regulator or customer could challenge: you need a human decision on record, not a model's opinion of a model.
This is the line that keeps eval automation honest. Over 40% of agentic AI projects are projected to be cancelled by end of 2027 on cost, unclear value, and weak controls (Gartner). A judge used as cheap human-review-at-scale strengthens your controls; a judge used to rubber-stamp high-stakes decisions is one of the weak controls that gets projects killed.
The stack: where to run and store evals
You do not need to build the harness from scratch. Three tools cover most production setups, and the choice comes down to open-source control versus a managed workflow.
Langfuse is open source and self-hostable for free, with a cloud Hobby tier that includes 50k units per month, Core at $29 per month for 100k units, and Pro at $199 per month for unlimited history and higher limits (Langfuse). DeepEval is fully open source and 4.0 ships G-Eval as a first-class metric, which makes it the fastest way to put a calibrated judge into your test suite (DeepEval).
Braintrust runs a hosted model: a free tier with $10 in credits, 1 GB of processed data, 10k scores, and unlimited users, then Pro at $249 per month with 50k scores (Braintrust). Whichever you pick, the tool is the harness, not the judge. The rubric, the golden set, and the calibration are still yours to own, and they are what determines whether the scores mean anything. For a deeper split on hosting and storing evals, see Langfuse vs LangSmith for production.
Is LLM-as-a-judge reliable enough for production?
Yes for triage, monitoring, and regression testing on open-ended text, once you have calibrated it against a human-labeled golden set and stated its agreement rate. No as the sole automated gate for safety-critical, compliance, or exact-correctness decisions, where a human approval or a deterministic check belongs instead.
Which model should do the judging?
Start with the strongest model you can afford for the best human alignment, then test whether a cheaper model still clears your agreement bar on the golden set. For pairwise scoring, avoid letting a model judge output from its own family, since self-preference bias inflates the result.
Binary labels or a 1-10 score?
Binary or a defined 1-5 Likert scale where each level is written out. Fine-grained numeric scores do not reproduce across raters or runs, so a 1-10 judge gives you precision you cannot trust. Split multiple concerns into separate single-axis judges rather than one blended score.
How big does the golden set need to be?
Big enough to estimate judge-human agreement with confidence, typically a few hundred labeled examples that span your real failure modes, not just happy-path cases. Grow it over time by adding every production example where the judge and a human disagreed.
Do chain-of-thought or few-shot prompts fix the bias?
They help alignment but do not remove position, verbosity, or self-preference bias. Reasoning steps improve the verdict; you still need order-swapping, length controls, a cross-family judge, and calibration to know how much to trust the number.
Start the Rescue
Agent scores you cannot trust are usually an eval problem, not a model problem. We calibrate your judges against a golden set, control the biases, and rebuild the eval gate so your metrics track reality.
Jul 13, 2026
