How attention blends meaning

What this lesson shows

Q·K → scores → softmax → weighted sum of Values.

The question it opens with

How do raw scores become a blend of meanings?

The walkthrough, in the lesson's own words

• Drag Sharpness to sharpen or blur the blend, then watch the shares redistribute.
• New job for the same sentence: raw scores become one blend that adds to 100%.
• Your turn to guess before the grid reveals it.
• Click any word's row. Drag the slider to sharpen or blur its focus.
• Same sentence “the cat drinks milk”, new job: each word now rebuilds its meaning from the words around it. Softmax turns raw scores into one blend that always adds to 100%.
• A word that knows its full context is exactly what lets the model predict what comes next. (Next lesson: why a word isn't allowed to peek at the words that come after it.)
• After softmax blends them, which word does it lean on most?
• Low = laser focus on one word. High = attention spreads out evenly.
• Each share scales that word's Value, the meaning it carries (from 3.2); add them up and you get
• Each row's bar shows where that word looks. Drag Sharpness; the shares always add to 100%.
• Every word rebuilds its meaning from the words it attends to.
• Attention just picks the one best word and ignores the rest.
• It blends. Softmax splits 100% across every word it can see, so the winner leans hardest but the others still add a little, and that mix is the word's new meaning.
• Combine this attention step with the other pieces you'll meet next, a small per-word network, plus a couple of stabilizing tricks, to make one transformer block, and repeat that block many times to get the engine behind ChatGPT, Claude, and the rest.
• Ask a model: the trophy didn't fit in the suitcase because it was too big, what was too big? It answers the trophy. What let it resolve it?
• Attention: when processing it, the model weighs every earlier word and leans hardest on trophy, blending that meaning in. Resolving pronouns, long-range references, and context-dependent words is attention deciding what to look at.
• These Q·K match scores from 3.2 were set by training, any size, just a pile of votes.
• Softmax pours those votes into one bar that's always 100%…

Key takeaway

See one word's meaning get updated by its context.

What you can do after this lesson

You can explain attention end to end: Q·K → softmax → weighted sum of Values.

Prefer reading first? This lesson has a companion explainer: Attention · Softmax

Where it leads: During training the model could cheat by seeing the future…

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.