LLM-as-a-judge

What this lesson shows

A strong model grades answers against a clear checklist: fast and scalable, but watch its biases.

The question it opens with

You have 10,000 open-ended answers to grade. Humans can't score them all.

The walkthrough, in the lesson's own words

• 10,000 free-form answers, no right string to match. Tap the scoring guide to grade one. What does the grading?
• A judge model scores each answer against a scoring guide, and picks a winner in A-vs-B.
• Answer B is genuinely better. Swap the read order, what's the judge's verdict? Guess first.
• Stack the mitigations and watch human agreement climb.
• A model can grade at scale, if you control its bias and check it against humans.
• The grader is itself a strong model: give it the question, the answer, and a scoring guide, and it returns a score in seconds, for cents. That's LLM-as-a-judge, no string match, no human panel needed.
• Pointwise scores one answer against a scoring guide; pairwise hands the judge two and asks which is better, often more reliable than absolute scores.
• This scoring guide is reference-free, no gold answer needed, just criteria. With a reference answer to compare against, it's reference-based.
• In A-then-B order the judge picks A, the worse one. Swap to B-then-A: what's its verdict?
• That's position bias, the judge just picks whatever it reads first. It's one of three that show up again and again:
• You don't need to label all 10,000, a few dozen human-graded items is enough to check and tune the judge.
• Order averaged, a strong grader, scoring guide calibrated to human labels, the judge now agrees with humans ~9 times out of 10. That's a judge you can ship behind your eval.
• Scalable grading, but only once you've checked it against the ground truth, which is exactly what the next lesson is about: labeled data.
• A team auto-scores its support replies with an LLM-as-a-judge, and the longer, hedging answers always win. Why, and what would you fix?
• Verbosity bias: the judge rewards length and surface thoroughness over whether the answer actually resolves the issue. Use a sharper scoring guide that scores resolution rather than length, and calibrate the judge against a few dozen human-graded items so its scores track ground truth instead of word count.
• Tap “Complete” to cover the edge case and watch the score recompute.
• The judge returns a verdict, but on what basis? Quality, or just reading order? The next step tests it.
• Measured against a held-out set of human-graded answers, your ground truth.
• Invented agreement numbers; the direction is real (each fix helps), the exact points are not measured.

Key takeaway

A model can grade at scale, if you control for its biases and check it against humans.

What you can do after this lesson

You can explain LLM-as-a-judge, its biases, and how to make it trustworthy.

Prefer reading first? This lesson has a companion explainer: LLM as a judge

Where it leads: Both evals and judges need a source of truth: labeled data.

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.