Hallucination

What this lesson shows

It optimizes plausibility, not truth; connect to RAG as mitigation.

The question it opens with

Why does a confident model make things up?

The walkthrough, in the lesson's own words

• Flip the toggle to a fact it can't know. Watch the odds go flat.
• Start with a fact it really knows. Watch the odds.
• Now flip to a fact it can't know. Same machine, different odds.
• Which question makes the model make something up?
• Eldermoor is invented, it was never in the data.
• See how flat that is? No answer was ever in the data, so no word wins, yet the model still picks one and states it as fact.
• One clear winner: this fact was all over training. Now flip to the made-up village to see the odds go flat.
• A model has no confidence dial, it always sounds fluent, even with nothing to go on.
• At every step it just hands back odds for the next word (the distribution from 4.1) and picks one, it can say the words “I'm not sure,” but that phrase is just another guess, not a readout of how sure it actually is.
• Its top guess is only 21%, barely ahead of the rest, yet it states it as fact, because it has no way to say “I'm only 21% sure.”
• Current research frames hallucination as an incentive problem (OpenAI's 'Why language models hallucinate', 2025). Most AI benchmarks are graded pass/fail: a confident guess sometimes scores a point, 'I don't know' never does. Train and pick models by those scoreboards and you breed good test-takers, models that learned that guessing beats abstaining. The proposed fix is to change the grading so a calibrated 'I don't know' earns credit.
• The healthy case: one word wins big, the rest barely register. Next, ask something it can't possibly know, and watch that spike collapse.
• A missing fact is the clearest case, nothing is known, so every plausible word is wrong, but the same machine can also garble or misattribute facts that were in its training. Fixes: hand it real sources (RAG, from 5.2), give it tools, or train it to say “I don't know.”
• Two more everyday causes: an ambiguous question invites a confident guess at what you meant, and in a RAG setup weak retrieval can hand the model the wrong chunk, so it answers fluently from the wrong source (how retrieval misses is lesson E.15's whole subject).
• Often, yes. Most training data is English, so there's thinner ground for other languages, and names collide across regions: 'tortilla' is a potato omelette in Spain but a flatbread in Mexico, and a 'Dr. García' could be any of thousands. Less data plus more ambiguity means more confident guessing, so ground it with sources and be extra skeptical of specific names, dates, and places in Spanish.
• When a model makes something up, its training data must have been wrong or out of date.
• It optimizes for plausible-sounding tokens, not truth, so it can fabricate confidently even with perfect data. The fix is to ground it with retrieval and verify specific claims, not to 'correct its facts'.
• A model confidently cites a court case, with a docket number, that turns out not to exist. Why does a fluent answer like this happen, and what should you do about it?
• The model has no confidence dial: it returns the most plausible next words and states them as fact, even when no real answer was in its training. So treat any specific claim, names, numbers, citations, as unverified until you check it against a real source.

Key takeaway

Understand the failure mode mechanically.

What you can do after this lesson

You can explain why models hallucinate: they pick plausible words, not true ones.

Prefer reading first? This lesson has a companion explainer: Hallucination

Where it leads: So how do you stop it? Pin the model to real sources before it answers.

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.