Predict the next word
Language needs context.
The cat
You've reached the end of the free preview. The next lesson needs your pass:
How much context? Condition on the previous word.
2.2 A first guess: just the last wordBuilds on1.1Turning words into numbers
Common questions
What is "Predict the next word" about?
What makes this tricky?
What will I be able to do after this lesson?
What comes next?
What this lesson shows
Language needs context.
The question it opens with
Predict the next word using only single-word frequencies: it's garbage.
The walkthrough, in the lesson's own words
- Strip away meaning: this toy model knows only word frequency. Tap the word it picks.
- Same list, every step. Watch it re-pick the top word.
- The model looks up an answer it stored somewhere.
- It only predicts the next token from patterns, so it can sound sure and still be wrong.
- To say anything sensible, a model has to read the words around the blank, context.
- People say an LLM looked something up or knows a fact. Mechanically, what is it actually doing when it answers?
- Predicting the next token, over and over, from patterns in its training, not retrieving a record from a database. That's why it can sound confident and still be wrong: it's producing what's likely to come next, not reading a stored fact.
- Toy fit scores, invented to show the idea; the frequency ranking reflects real English averages.
- “the” is just the most common word, so it wins regardless of the sentence.
- Let it read “The cat” and “drinks” shoots to the top, even though it’s rare.
- With no context, the bare frequency list wins and “the” always tops it.
- “drinks” actually fits after “The cat”, but this model never read “cat”. In lessons 1.2–1.3 a word became a vector, so the model could find which word fit. Take that away and only one thing is left: how often each word appears. With no context, the single most common English word wins every blank.
Key takeaway
You feel firsthand why context matters: ignore it and the model writes nonsense.
What you can do after this lesson
You can explain why word frequency alone fails and context is essential.
Check yourself: Why is word frequency alone a terrible way to predict text?
- It ignores context entirely(correct)
- It's too slow
- It needs a GPU
- It overfits
Frequency picks the same common word every time; it ignores the sentence so far, which is exactly what decides the next word.
Prefer reading first? This lesson has a companion explainer: Next-token prediction
Where it leads: How much context? Condition on the previous word.
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.