Lesson 3 of 6 · 48 min

LLM-as-judge & its traps

The dominant reference-free metric is itself a biased instrument. Pointwise vs pairwise by criterion shape, the position/verbosity/self-preference biases with measured magnitudes, the prompt mechanics that mitigate them, and why a judge is a probabilistic sensor you must calibrate.

A measurement instrument, not a metric

LLM-as-judge is how you scale reference-free evaluation — but it is a measurement instrument, and instruments must be calibrated. A judge that agrees with humans 85% of the time while preferring verbose answers 90% of the time will silently fail to flag the regression when you add a verbosity penalty. This lesson is the mechanics: when to score one output vs compare two, the named biases with their measured magnitudes, the prompt structure that suppresses them, and the validation that turns "the judge said 88% pass" into a number with honest error bars.

Why a judge works at all: an LLM prompted with a rubric and a candidate output returns a pass/fail or a winner, and on calibrated benchmarks this tracks humans surprisingly well — GPT-4 hits ~85% agreement with human experts on MT-Bench, exceeding the 81% human-human ceiling, and Chatbot Arena’s judge panel clusters at 83–87%. But those aggregate numbers are seductive and partly misleading: the same judges show ~50–70% position bias, >90% verbosity bias, and 10–25% self-preference. High aggregate agreement can coexist with severe per-instance systematic bias — which is the whole reason this lesson exists.

Eugene Yan on Using LLMs as JudgesEugene Yan / Jason Liu

Pointwise vs pairwise: match the comparator to the criterion

The choice is dictated by the shape of the criterion, not by preference. Pointwise judges score a single response against the question and a rubric — best for objective, definitional criteria ("does it cite a source?", "is it non-toxic?", "is it on-policy?"), because the judge can check a binary fact. They degrade on subjective criteria, where the judge must internalise an absolute scale it has no anchor for. Pairwise judges compare two responses and pick a winner (or tie) — best for subjective criteria ("which is more concise?", "which is more helpful?"), because relative comparison is far more concrete for an LLM than absolute scoring, and it mirrors the unit of a real business decision (A vs B).

The senior nuance: pairwise gives power but only a relative ranking — it cannot tell you whether both systems are bad, only which is better. So pairwise is the tool for "is the new prompt better than the old?", while pointwise (with a calibrated threshold) is the tool for "is this answer acceptable in absolute terms?". One more rule the research is firm on: one evaluator per dimension, never a "God Evaluator" that rates five things in one prompt — the multi-purpose judge is materially worse on each individual dimension because its calibration budget is split across outputs. Interview angle. "Pairwise or pointwise here?" → objective/factual → pointwise; subjective/stylistic → pairwise; and "the better of a pair might still be defective," so don’t use pairwise to certify absolute quality.

code

1Pointwise vs pairwise -- by criterion shape23  Criterion type        Use         Why4  -------------------   ---------   -------------------------------------5  Faithful / on-policy  pointwise   binary fact the judge can check6  Toxic / refuses       pointwise   absolute threshold, definitional7  More helpful?         pairwise    relative comparison is concrete8  More concise / tone   pairwise    subjective; absolute scale is noisy9  "Is this good?"       pointwise   needs a calibrated PASS threshold10  "A vs B for ship?"    pairwise    mirrors the business decision1112  Pairwise = power on subjective dims, but only a RELATIVE verdict13  (both can be bad). One evaluator per dimension -- never a God Evaluator.

The three biases that travel together

Three biases are independently measured and must be designed around — disclosing them is not enough. Position bias: in pairwise settings gpt-3.5 picks the first answer ~50% of the time and claude-v1 ~70%, even controlling for quality; it stems from the autoregressive formulation favouring earlier-positioned tokens. Verbosity bias: both gpt-3.5 and claude-v1 prefer the longer response >90% of the time even when it isn’t better — length is a learned proxy for quality. Self-preference (self-enhancement): GPT-4 scores its own outputs ~10% higher and claude-v1 ~25% higher; the leading hypothesis is that the bias lives in perplexity — a model prefers text it would itself assign high probability to. These three interact, so fixing one while ignoring the others is the standard failure.

code

1Judge biases with measured magnitudes (design around, don't disclose)23  Bias              Magnitude (reported)        Mitigation4  ---------------   -------------------------   -----------------------------5  Position          gpt-3.5 ~50%, claude ~70%   swap A/B, run twice, flip=TIE6  Verbosity         both >90% prefer longer     length-budget criterion; blind7  Self-enhancement  gpt-4 +10%, claude-v1 +25%  judge family != generator family8  Style/confidence  "polished" favored           strip formatting or add rubric910  High aggregate agreement (85% MT-Bench) can hide all of these per-instance.

Beyond the classic three, the CALM framework ("Justice or Prejudice?") catalogues 12 bias vectors — authority, bandwagon, distraction, sentiment, diversity, chain-of-thought, refinement-aware, and more — and scores judges with metrics like Robustness Rate (does the verdict change after bias injection?) and Consistency Rate (is it stable across identical repeats?). The finding that matters for model selection: Claude-3.5 was generally the most resilient overall, but top-tier models still showed unexpected weaknesses on individual axes (e.g. GPT-4-Turbo wobbled on sentiment when judging emotional responses; most models were swayed by fake "book" and "quote" citations — authority bias). The lesson: bias robustness is a per-model, per-axis property, not a brand attribute — profile the specific judge you plan to ship, and re-profile it on every model upgrade.

Prompt mechanics that suppress the biases

The prompt structure is itself a mitigation. The converged template across Eugene Yan and OpenAI’s grader docs: (1) an identity/role anchor ("You are an expert evaluator of [domain]"); (2) the question and response(s) inside clearly-labeled delimiters — Eugene’s production pattern uses XML tags like <control> / <treatment>, because LLMs pattern-match off token-boundary cues and untagged concatenation makes the judge silently mix which output is which on long inputs; (3) a single definitional criterion; (4) an output-format constraint ("respond with one of PASS, FAIL, TIE"). For position bias specifically, run the pairwise comparison twice with the order swapped, and if the winner flips, mark it a TIE — the cheapest, fastest mitigation and the canonical interview answer.

python

1# Position-bias mitigation: judge twice with swapped order; a flip means TIE.2def pairwise_judge(question, a, b, judge):3    def ask(first, second):4        prompt = (5            "You are an expert evaluator. Pick the more helpful answer.\n"6            f"{question}\n"7            f"{first}\n{second}\n"8            "Reason briefly, then reply with exactly: ANSWER_1, ANSWER_2, or TIE."9        )10        return parse_verdict(judge(prompt))      # show reasoning BEFORE the verdict11    v1 = ask(a, b)                                # A in slot 112    v2 = ask(b, a)                                # A in slot 2 (swapped)13    if v1 == "ANSWER_1" and v2 == "ANSWER_2":14        return "A"                                # A won in BOTH orders15    if v1 == "ANSWER_2" and v2 == "ANSWER_1":16        return "B"                                # B won in both orders17    return "TIE"                                  # verdict flipped -> position bias18# Keep this in the eval library, not the prompt -- prompt-only mitigations rot silently.

Two more mechanics worth knowing. Chain-of-thought before the verdict: ask the judge to show its reasoning then output the score, so the reasoning anchors the verdict rather than rationalising a snap judgement — but note CoT does not immunise against the biases above, so it complements swap-and-resolve, it doesn’t replace it. And include a reference answer when you have one: passing the gold answer into the judge prompt turns that call into a reference-based check and sharply improves agreement (Prometheus-style judges degrade most when references are removed) — which is why reference-based grading is preferred wherever gold coverage exists, and reference-free is the production fallback where it’s thin.

The judge is a probabilistic sensor — calibrate it

The discipline that makes a judge trustworthy: treat it as a sensor with a known true-positive rate (TPR) and true-negative rate (TNR), measured against a held-out human-labeled set, and bias-correct the headline. The inversion is one line — the rate the judge reports is p = π·TPR + (1−π)·(1−TNR), where π is the true failure rate. Make it concrete: a judge with TPR=0.9, TNR=0.85 grading a system whose true failure rate is 10% reports 0.10·0.9 + 0.90·0.15 = 0.225 — 22.5%, more than double the truth, because false positives on the 90% pass class (13.5 points) swamp the 9 points of genuine failures it catches. Solve the same equation for π to back out the true rate from the measured one. The lesson: an uncalibrated judge’s headline number can be off by 2× on an imbalanced class, easily enough to flip a shipping decision. Every metric a judge produces is therefore an estimate that needs a confidence band, not a point estimate. If your team cannot state the judge’s TPR/TNR for the current rubric, the metric is not ready to use. This is the same calibration mindset you’d apply to any noisy classifier — applied to the classifier you built out of a prompt.

Calibration is not a one-time event — it’s an alignment loop. Shreya Shankar’s "Who Validates the Validators?" (and LangChain’s productised Align Evals) make the mechanism concrete: collect human corrections on a representative sample, build few-shot examples from those corrections into the judge prompt, and track judge-vs-human agreement on a rolling window — so the gold-labeled examples update the judge, not merely measure it. Prompt iteration alone won’t close the gap between a technically-correct evaluator and a reliable one; systematic alignment to human corrections will. Interview angle. "Your judge disagrees with humans on 12% of cases — what do you do?" → measure TPR/TNR on a holdout, error-analyse the false positives and negatives, fold the disagreements in as few-shot anchors, and re-validate — not "retrain the judge" or "trust GPT-4."

Single judge, a panel, or a fine-tuned one?

Once the judge is calibrated, three architectures trade cost for robustness. A single frontier judge is simplest and the default for low volume. A panel of judges (multiple model families, majority vote) reduces single-model bias and variance — useful when one family’s self-preference or sentiment quirk would otherwise dominate, and it raises agreement on contested cases — but it multiplies cost linearly and can hit the eval-bill cliff (L5). A fine-tuned small judge (an 8B model aligned to your domain rubric) is the scale answer: it can be 30–50× cheaper than calling a frontier model every time while matching or beating it on your rubric, because domain alignment beats raw capability on a narrow task. The senior pattern at volume: a cheap fine-tuned judge on the bulk of traffic, a frontier judge (or panel) reserved for the ambiguous residual and for re-calibration.

One operational trap specific to judges: a judge model upgrade is a silent re-calibration event. The CALM finding is that a major model version can shift the bias profile even when headline benchmark numbers look similar — so swapping your judge from one version to the next, or letting a provider auto-upgrade it, can move every metric it produces without any change to your system. The defense mirrors the model-upgrade gate for the generator (L5): re-run the bias scorecard and re-measure TPR/TNR against the human holdout whenever the judge changes, and pin the judge version alongside the generator version. Interview angle. "Your judge auto-upgraded last week and the pass rate jumped 4 points — real improvement?" → no — treat it as judge drift, re-profile biases and re-calibrate before believing the number.

articleEvaluating the Effectiveness of LLM-Evaluators (the bias survey)Eugene Yan paperJustice or Prejudice? Quantifying Biases in LLM-as-a-Judge (CALM)Ye et al. (arXiv)articleHow to Calibrate LLM-as-Judge with Human Corrections (Align Evals)LangChain videoHow to align your LLM judge for better evaluationsWeights & Biases

Checkpoint

You’re comparing two prompt variants on subjective helpfulness with a pairwise GPT-4 judge. A reviewer warns the result may be an artifact. What’s the most important guard to add?

ARun the comparison twice with the answer order swapped and treat a flipped winner as a TIEBRaise the judge’s temperature so it considers more optionsCAsk the judge to be objective in the system prompt

Checkpoint

Your generator is GPT-4 and you set up a GPT-4 pairwise judge to compare it against a competitor model. What’s the design flaw?

ANo flaw — GPT-4 is the strongest judge available, so use itBSelf-enhancement bias — GPT-4 favours its own outputs (~+10%); use a judge from a different model family than the generatorCGPT-4 is too expensive to judge at scale, so use a smaller model

Checkpoint

A PM wants one LLM judge that rates each answer on helpfulness, faithfulness, tone, conciseness, and safety in a single prompt to save cost. What’s the senior recommendation?

ABuild it — a single capable judge can handle five dimensions and it’s cheaperBUse one pairwise judge for all five dimensions at onceCOne evaluator per dimension, and choose pointwise for objective dims (faithfulness, safety) and pairwise for subjective ones (tone, conciseness)

Checkpoint

Your reference-free faithfulness judge consistently passes longer answers. You add a "be concise" line to the judge prompt but the bias persists. Best next move?

ATrust the judge — longer answers are probably genuinely more thoroughBAdd an explicit length-budget criterion (or length-blind the inputs), and validate against a human-labeled holdout to confirm the bias is goneCSwitch the judge to a larger model

Checkpoint

Your judge reports 88% pass on a production sample. A PM asks "is that the real failure rate?" What’s the correct answer?

AYes — 88% pass means a 12% real failure rateBNot directly — invert the judge’s TPR/TNR (measured on a held-out human-labeled set): the reported rate is p = π·TPR + (1−π)·(1−TNR), so the true failure rate π can differ materially from 12% — easily enough to flip a decision on an imbalanced classCIt’s unknowable — LLM judges can’t be trusted at all

Interview prep

LLM-as-judge rounds test whether you treat the judge as a calibrated instrument with known failure modes. Interviewers probe four things: do you pick pointwise vs pairwise by criterion shape, can you name the biases with magnitudes and their mitigations, do you know the prompt mechanics (XML tags, swap-and-resolve, CoT-before-verdict), and do you calibrate against humans with TPR/TNR rather than trusting the raw rate. Name numbers.

01“Pointwise or pairwise?” → objective/definitional → pointwise; subjective/stylistic → pairwise; pairwise gives only a relative verdict (both can be bad).
02“Name the judge biases.” → position (gpt-3.5 ~50%, claude ~70%), verbosity (>90% prefer longer), self-preference (gpt-4 +10%, claude-v1 +25%).
03“Fix position bias?” → run twice with swapped order; if the winner flips, mark TIE — keep it in the eval library, not the prompt.
04“Fix self-preference?” → use a judge from a different model family than the generator; never grade a family’s outputs with its own model.
05“Why not one judge for five dimensions?” → the God Evaluator is worse per dimension; one evaluator per dimension, calibration budget undivided.
06“Does chain-of-thought remove bias?” → no — it makes verdicts legible, not unbiased; pair it with structural mitigations.
07“Is the judge’s 88% the real rate?” → correct for TPR/TNR on a holdout; the raw rate ignores the judge’s measurement error.
08“Judge disagrees with humans 12% — next step?” → error-analyse the FPs/FNs, fold disagreements in as few-shot anchors, re-validate — alignment loop, not retrain.

Going deeper, the curveballs: "reference-based or reference-free judging — when each?" (reference-based when gold answers exist, since judges degrade most when references are removed; reference-free in production where gold coverage is thin); "your judge upgrades from one model version to the next — what breaks?" (the bias profile can shift even when benchmark numbers look similar, so re-run the CALM-style scorecard and re-calibrate TPR/TNR); and "how do you bias-test a brand-new judge before shipping?" (profile position, verbosity, self-preference, authority, and sentiment on injected examples, and gate promotion on the scorecard). Tie each to a measured magnitude.

Could you design a calibrated LLM judge — comparator, bias mitigations, prompt structure, TPR/TNR validation — and defend each choice?

New to itGetting thereConfident

Takeaways

A judge is a measurement instrument — high aggregate agreement (85% MT-Bench) can hide severe per-instance bias.
Pick pointwise for objective/definitional criteria, pairwise for subjective ones; one evaluator per dimension.
Design around the measured biases: position (~50–70%), verbosity (>90%), self-preference (+10–25%).
Mitigate structurally: swap-and-resolve for position, length-budget for verbosity, cross-family judge for self-preference.
Chain-of-thought makes verdicts legible, not unbiased; include a reference answer when you hold gold.
Calibrate the judge as a sensor — measure TPR/TNR on a human holdout and bias-correct the headline rate.

Next: faithfulness & relevance — Ragas RAG metrics, and measuring judge-human agreement with Cohen’s kappa.

Sources

Free to read · better with Enzo

Learn it with Enzo

Save your progress, answer the checkpoints, and let Enzo quiz you on what you just read.

Continue with Enzo Sign up free