Why context changes everything
To finish 'The cat drinks ___', a token must look back at 'cat'. Bigrams see only the previous word.
The idea inside
Tokens must look at other tokens.
After this lesson
You can explain why a model must let words look at other words (attention).
Where it leads
How does a token decide what to look at?
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
Tokens must look at other tokens.
The question it opens with
To finish 'The cat drinks ___', a token must look back at 'cat'. Bigrams see only the previous word.
The walkthrough, in the lesson's own words
- Tap the word you think fills the blank. A bigram sees only “drinks”.
- Which earlier word would unlock the right answer?
- Watch the blank's neighbor reach back to that word.
- That reaching-back is the whole idea of attention.
- Right, only “cat” tells us who's drinking, and cats drink milk. That's the clue a bigram never sees.
- A bigram remembers one word. Attention lets every word reach back across the whole sentence and weight what it finds, “drinks” found “cat” two words away. That reach, run in parallel over long passages, is what made modern language models possible.
- Nobody tells the model that a verb should look back at its subject. It learns which words to pull while training to predict the next word (Act 2), the same game you played there.
- You paste a long email thread and ask a model to summarize it. It correctly says “she” in the third message means the manager named at the very top. How does it reach back that far?
- That is attention: when it processes “she”, the model looks back across every earlier word and weights the manager hardest, pulling that meaning in. The same reach that linked “drinks” to “cat” here is what lets it track who and what across pages of context.
- You knew a cat was drinking. The bigram never sees “cat”, so it can't.
- A fair guess: with only “drinks” to go on, every word here fits.
Key takeaway
Motivates attention.
What you can do after this lesson
You can explain why a model must let words look at other words (attention).
Check yourself: Why must a token be able to look at the other tokens?
- A word's meaning depends on the words around it(correct)
- To count how many tokens there are
- To sort the tokens alphabetically
- To save memory
The same word means different things in different sentences ('bank' by 'river' vs 'money'), so each token has to read its neighbours. That is what attention does.
Where it leads: How does a token decide what to look at?
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.