Plain-language explainer
LLM evals, explained
What are evals, and how do teams know an AI feature actually works?
An eval is a repeatable test for an AI feature: a set of inputs, and a way to score whether the outputs are good enough. Because models are non-deterministic and 'looks fine' does not scale, teams build evals to catch regressions before users do. Scoring can be exact checks, rubrics, or another model acting as a judge. The hard part is keeping evals honest: a frozen offline set can go stale or leak into training, so production and adversarial tests catch what it cannot.
Last reviewed
Reading is the slow way. Start with a free lesson you can operate right now.
Start free: From the internet to your answer →Free, no code, no signup.
Then go deeper: Evals: proving it works Locked
What people get wrong
- A high offline score means it is safe to ship. The set can be stale or contaminated; watch live traffic too.
- Evals are just unit tests. They score fuzzy quality, often with rubrics or a model judge, not exact equality.
- A model judge is unbiased. Judges have biases and need calibration against human ratings.
Where you see it in real products
- Teams gate releases on an eval suite, like tests in CI.
- Online evals score a sample of real traffic after launch.
- Red-team evals probe for failures and prompt injection before users find them.
Common questions
- Why not just use public benchmarks?
- They measure general ability on tasks that are not yours, and popular ones leak into training data over time. A small set of your own real examples with known good answers tells you more about your product than any leaderboard.
- How many test cases do I need to start?
- Twenty real ones beat none, and beat two hundred invented ones. Take them from actual traffic, including the failures that annoyed you, and add a case every time something breaks in production.
- What do you actually measure?
- Whatever failure would hurt. Exact match where there is a right answer, a checked rule where format matters, a rubric score where quality is a judgement call. Pick the cheapest check that would have caught your last real bug.
Related explainers
More in Building on it, and trusting it
One idea at a time, in your inbox
New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.
We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.
Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.