What is attention actually doing to a word?

For every word it scores how relevant each other word is, then mixes those words together in the scored proportions. The result replaces that word's representation with a version coloured by its context, which is how bank ends up meaning different things in a river sentence and a money sentence.

What is multi-head attention?

The same mechanism run several times in parallel with different learned weights, then combined. Each head can specialise, one tracking the grammatical subject, another tracking a reference made twenty words ago, so one layer follows several kinds of relationship at once.

Why is attention expensive on long inputs?

Every token compares itself to every other token, so that work grows with the square of the sequence length. Doubling the input roughly quadruples it. That is why long context is a hard engineering problem rather than a setting someone forgot to turn up.

Plain-language explainer

Attention, explained

What does the attention mechanism do in a transformer?

Attention lets each word look at the other words in the sentence and decide which ones matter for it right now. In 'the trophy did not fit in the suitcase because it was too big', attention is what tells the model that 'it' refers to the trophy. Each position gathers a weighted blend of the others, leaning hardest on the ones that fit. This is how a model handles pronouns, long-range references, and the way a word's meaning shifts with its context.

Last reviewed July 25, 2026

Do not just read it. Operate the mechanism yourself in a short interactive lesson.

See it work: How attention blends meaning →

Free, no code, no signup.

What people get wrong

Attention reads strictly left to right. It can weigh every earlier word at once, not just the previous one.
It is keyword matching. It is a learned, weighted blend of meaning, not exact-word lookup.
More attention heads always means better. Heads specialize, and past a point you get diminishing returns.

Where you see it in real products

Every modern chat and coding model is built on stacked attention layers.
Long-document understanding depends on attention linking distant parts.
Quality on pronouns, code references, and citations comes from attention working well.

Common questions

What is attention actually doing to a word?: For every word it scores how relevant each other word is, then mixes those words together in the scored proportions. The result replaces that word's representation with a version coloured by its context, which is how bank ends up meaning different things in a river sentence and a money sentence.
What is multi-head attention?: The same mechanism run several times in parallel with different learned weights, then combined. Each head can specialise, one tracking the grammatical subject, another tracking a reference made twenty words ago, so one layer follows several kinds of relationship at once.
Why is attention expensive on long inputs?: Every token compares itself to every other token, so that work grows with the square of the sequence length. Doubling the input roughly quadruples it. That is why long context is a hard engineering problem rather than a setting someone forgot to turn up.

Related explainers

Transformers Query, Key, Value Embeddings

One idea at a time, in your inbox

New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.

We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.

Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.