Why it can't peek ahead
The model could peek at future words during training.
The idea inside
Mask the upper triangle to −∞ before softmax.
After this lesson
You can explain causal masking: a word sees only itself and earlier words.
Where it leads
One head sees one kind of relationship: what about others?
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
Mask the upper triangle to −∞ before softmax.
The question it opens with
The model could peek at future words during training.
The walkthrough, in the lesson's own words
- When the model writes, it predicts one word at a time, so as it decides what comes after “drinks”, the word that will come next (“milk”) hasn’t been written yet. Which words must “drinks” be stopped from reading?
- Flip the mask on and off. Watch the future cells (top-right) switch off.
- No peeking: a word sees only itself and what came before.
- Before you flip it: “drinks” currently leans some of its attention on “milk”. Hide “milk”, where does that attention go?
- Right, softmax always re-splits the shares so they add up to 100%. With “milk” at −inf, the survivors re-share the full 100%, just like the blend in 3.3.
- Not quite. Softmax always sums to 100% (the blend from 3.3). Hiding “milk” doesn’t leave a hole, its share is re-split among the words “drinks” can still see. Toggle below and watch the bars.
- Causal mask is on. Tap to allow peeking at the future.
- Peeking is allowed. Tap to turn the causal mask on.
- Each word attends only to itself and the words before it, so the only way to get the next word right is to truly predict it.
- The fix is brutally simple: before softmax, set every “future” score to minus infinity, so its weight becomes zero.
- With the future hidden, this is the same job the model does at inference, where the future doesn’t exist yet.
- Remember the next-word loss you drove in 2.3? The mask is what keeps that game honest: the model has to guess “milk” without being allowed to peek at it.
- You watch a chatbot stream its reply one word at a time, left to right, never going back to revise. Why can’t it just write the whole sentence at once and tidy it up?
- The causal mask: each position only ever sees itself and the words before it, so the model is trained to predict the next word from the past alone. That left-to-right, one-token-at-a-time generation you see is the mask doing its job.
Key takeaway
No peeking at the future.
What you can do after this lesson
You can explain causal masking: a word sees only itself and earlier words.
Check yourself: What does causal masking accomplish?
- A word can only see itself and earlier words(correct)
- It speeds up training
- It removes duplicate tokens
- It adds positions
Masking hides future words during training, so the model learns to predict from the left only, the same way it generates one word at a time.
Where it leads: One head sees one kind of relationship: what about others?
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.