Multi-head attention

What this lesson shows

Several heads attend to different relationships in parallel.

The question it opens with

One attention pattern can't track a word's subject and its neighbors at once.

The walkthrough, in the lesson's own words

• “drinks” must catch its subject AND its neighbors at once. Tap a head to try it, can one map do both?
• Meet the fix: several heads, each watching one kind of relationship.
• Your turn. Switch heads, each lights up a different pattern, same sentence.
• All heads run at once, each emits a vector. Concatenate them, then one projection mixes them into the layer’s output.
• Neither single map does both: grammar catches the subject but drops the neighbors; neighbors does the reverse. The fix: run several patterns side by side.
• The numbers in these vectors are invented; the merge steps (weighted sum, concatenate, project) are the real mechanism.
• Concatenated, lined up, not averaged, then one projection (W_O) mixes the heads into a single richer vector. Nobody assigns a head its job: every head starts random, and training pushes each toward a different pattern.
• The heads get averaged together into one blended attention map.
• They are lined up end to end, not averaged. Each head keeps its own output, and one projection then mixes them into a single richer vector.
• W_O is a learned weight matrix, the same kind that built Query, Key and Value from an embedding. Nobody sets its numbers by hand; training does, along with the grids inside every head.
• When researchers peek inside a model, they find one head that tracks subject-verb agreement and a different one that links quotes to who said them. Why are these jobs split across separate heads?
• That is multi-head attention: each head learns its own relationship pattern, and they all run in parallel on the same sentence. No single head can catch grammar and speaker and topic at once, so the model spreads the work across dozens of heads.
• Keep flipping between the maps, each catches one relation but never both.
• Each “head” is its own attention pattern, all on the same sentence.
• Each head finishes the attention step, a weighted sum of its own Values, and emits one output vector:

Key takeaway

One head for grammar, another for nearby words.

What you can do after this lesson

You can explain multi-head attention: parallel heads, each a different pattern.

Where it leads: What turns a blended vector into the next layer's input?

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.