What biases does an LLM judge have?

Three that show up repeatedly: a preference for longer answers, a preference for the first option presented, and a mild preference for text produced by a similar model. Randomising the order of the answers and capping length in the rubric removes most of the damage.

How do I know if my judge is any good?

Grade a sample by hand and measure agreement with the judge. If they disagree on a fifth of cases the judge is measuring something other than what you wanted, and the fix is almost always a sharper rubric with concrete examples of each score.

Should the judge be the same model being tested?

Prefer a different one where you can. A model scoring its own output is the case with the strongest self-preference effect, and using a separate model or an older version makes the comparison harder to fool.

Plain-language explainer

LLM as a judge, and when to trust the verdict

What does LLM as a judge mean, and does it work?

Using a model to grade another model's output against a written rubric, instead of paying a human to read every answer. It is fast, cheap and consistent enough to be the default in most eval pipelines. It also inherits the judge's blind spots: it rewards long, confident, well-formatted answers, it can prefer text from a model like itself, and it favours whichever answer it saw first. Calibrate it against human labels before you trust it.

Last reviewed July 25, 2026

Reading is the slow way. Start with a free lesson you can operate right now.

Start free: From the internet to your answer →

Free, no code, no signup.

Then go deeper: LLM-as-a-judge Locked

What people get wrong

A judge model is objective. It is another predictor with learned preferences, and those preferences show up as consistent, measurable biases.
A high judge score means users are happy. It means the answer matched a rubric. Whether the rubric matches what users want is a separate question you have to check.
You need a frontier model to judge. For narrow checks a small model with a tight rubric often matches a large one at a fraction of the cost.

Where you see it in real products

Regression suites that score every model upgrade before it ships.
Automatic content filters and quality gates on generated output.
Preference data collection, where a judge stands in for human raters at scale.

Common questions

What biases does an LLM judge have?: Three that show up repeatedly: a preference for longer answers, a preference for the first option presented, and a mild preference for text produced by a similar model. Randomising the order of the answers and capping length in the rubric removes most of the damage.
How do I know if my judge is any good?: Grade a sample by hand and measure agreement with the judge. If they disagree on a fifth of cases the judge is measuring something other than what you wanted, and the fix is almost always a sharper rubric with concrete examples of each score.
Should the judge be the same model being tested?: Prefer a different one where you can. A model scoring its own output is the case with the strongest self-preference effect, and using a separate model or an older version makes the comparison harder to fool.

Related explainers

LLM evals Hallucination LLM limits

More in Building on it, and trusting it

Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.