Boss: repair broken attention
You've watched tokens decide what to look at. Could you fix one that looks at the wrong word?
The idea inside
Each focus token's attention beam points at the wrong word. Reroute it to the word it really depends on, even when the break is a peek at the future.
After this lesson
You can spot a misrouted attention beam, pick the token it should attend to from the Q·K match, and respect the causal mask.
Where it leads
Attention is wired and stacked. Now the model has a ranked list of next words, so how does it pick one?
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
Each focus token's attention beam points at the wrong word. Reroute it to the word it really depends on, even when the break is a peek at the future.
The question it opens with
You've watched tokens decide what to look at. Could you fix one that looks at the wrong word?
The walkthrough, in the lesson's own words
- Attention is just a beam. You repaired where each token looks.
- Repaired. The beam snapped to the right word and the blend re-normalised onto it.
- This token's attention points at the wrong word. Tap the word it should attend to.
- Toy shares and Q·K scores, chosen to show the repair. A real head computes them from the Query and Key vectors, whose grids it learned in training.
- That word comes after the focus token. The causal mask blocks it. Pick an earlier word.
- Closer, but that word's Key does not match the Query. Re-read who the token refers to.
- The word you repaired to has the highest Key match, that is why it wins the blend.
- You rerouted each token's attention to the word it actually depends on.
- All three breaks are fixed, but some picks needed a retry. Repair them again to clear every beam on the first try.
- Attention is a routing decision. A token emits a Query (3.2), each word answers with a Key, the best match wins the softmax blend of Values (3.3), and a causal mask forbids looking ahead (3.4). Repairing the beam is doing that routing by hand.
- For each break you read who the focus token refers to, then sent its Query to the word with the matching Key. The softmax row re-normalised onto your pick, so the shares still added to 100%. The mask break showed the one hard rule: a word can never attend to a word that comes after it, no matter how good the match looks. Stack many of these routing decisions in parallel and you have multi-head attention (3.5).
Key takeaway
You rerouted attention by hand: matched a Query to the right Key and respected the causal mask.
What you can do after this lesson
You can spot a misrouted attention beam, pick the token it should attend to from the Q·K match, and respect the causal mask.
Where it leads: Attention is wired and stacked. Now the model has a ranked list of next words, so how does it pick one?
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.