Every closed-loop architecture leans on evaluation, and at production volume that means LLM-as-judge: models scoring the outputs of other models. Used well, it's the only economical way to measure judgment-dependent quality at scale. Used the way most teams first use it, it produces beautifully precise-looking numbers that measure almost nothing.
The difference is entirely in rubric design and calibration. A judge is a measurement instrument, and nobody trusts an instrument they haven't calibrated.
The bare-score trap
The first thing everyone tries: "Rate this response 1–10 for quality." The scores come back looking authoritative and are mostly noise — the same response can score 6 or 9 across runs, and the average mostly tracks fluency. A confident, well-structured, wrong answer outscores a terse correct one. You've built a machine for measuring how persuasive your agent sounds, which is the one thing an LLM never lacked.
Decompose, then demand evidence
Reliable judging looks like a checklist, not a vibe. Break "quality" into specific, individually-verifiable checks: Did the response address the customer's actual question? Quote the sentence where it does. Does the cited policy section exist and say what the response claims? Did the agent take every required action before closing? Binary or three-level verdicts per check, with the judge required to cite the evidence for each verdict. Evidence citation isn't decoration — it forces the judge to ground each verdict in the text and gives you something auditable when you disagree with a score. Aggregate the checks into the score, weighted by what actually matters for the task.
Calibration: the step that separates signal from theater
Here's the uncomfortable question for any judge pipeline: how do you know the judge is right? The answer has to be data you trust more than the judge. Two sources:
- Human-labeled sets. A few hundred runs labeled carefully by people who know the domain. Measure judge-human agreement per check; iterate on rubric wording until agreement is strong on the checks that matter; drop or redesign checks where humans don't even agree with each other.
- Outcome data. The judge said 9/10; the ticket reopened the next day. At volume, correlate judge scores with real outcomes — reopens, reverts, user retries. A judge whose high scores don't predict good outcomes is measuring the wrong thing, however plausible its reasoning reads. This is why outcome joining comes before evaluation in build order: outcomes are what you calibrate against.
Known biases, known countermeasures
Judge models have documented, repeatable biases. Verbosity bias: cap or normalize for length, and include checks that penalize padding. Position bias in pairwise comparisons: randomize order and average both directions. Self-favoritism: where feasible, judge with a different model family than the one being judged. Drift on ambiguous wording: freeze rubric versions, and re-run a calibration sample whenever the judge model itself updates — your instrument can drift just like your agent.
Version the rubric like code
One operational rule saves months of confusion: every score is stamped with the rubric version and judge model version that produced it. Change the rubric and your metric moves — without versioning you can't tell an agent regression from a measurement change. With it, rubric improvements are just another gated change in the loop.
Done this way, LLM-as-judge stops being a leap of faith and becomes what the loop needs: a sensor whose error bars you know. It will never be perfect — it doesn't need to be. It needs to be calibrated, consistent, and cheaper than the humans it replaces, and that bar is very reachable.