Why does a transformer need position information at all?

Attention is order-blind: it sees a bag of tokens, not a sequence. Without a position signal, the dog bit the man and the man bit the dog are identical inputs. Position is added so the model can tell them apart.

How do modern models encode position?

Most large models now use rotary embeddings, known as RoPE, which rotate the query and key vectors by an angle that depends on position. Relative distance then falls out of the maths naturally, and the scheme stretches to longer inputs better than the original fixed sine tables.

Is this why models get worse past their trained length?

Largely, yes. Beyond the positions seen in training, the encoding drifts into territory the model never learned to read. Most techniques for extending context work by rescaling or retraining those position signals rather than by changing attention itself.

Plain-language explainer

Positional encoding, explained

How does a transformer know word order, if attention looks everywhere at once?

Attention on its own treats a sentence as a bag of words: 'dog bites man' and 'man bites dog' would look identical. So models mix a position signal into every token's vector before attention runs. Each position gets its own mathematical stamp, and the stamp becomes part of the token's meaning, so 'first word' and 'seventh word' are different even for the same word. Many modern models use a rotating variant, RoPE, which encodes relative distance between words.

Last reviewed July 25, 2026

Reading is the slow way. Start with a free lesson you can operate right now.

Start free: How attention blends meaning →

Free, no code, no signup.

Then go deeper: How the model knows word order Locked

What people get wrong

Models read left to right the way people do. Attention sees every token at once. Order exists only because it was stamped in.
Word order is stored in the words themselves. The same word at two positions gets two different vectors, and only the stamp differs.
Position is a minor implementation detail. Stretching models to longer contexts is hard largely because the position scheme must stretch with them.

Where you see it in real products

Translation tools keep who-did-what-to-whom straight because position is part of meaning.
Code assistants track nesting and bracket order through the same signal.
Long-context model releases often name their position-scheme tricks, RoPE scaling among them, in the fine print.

Common questions

Why does a transformer need position information at all?: Attention is order-blind: it sees a bag of tokens, not a sequence. Without a position signal, the dog bit the man and the man bit the dog are identical inputs. Position is added so the model can tell them apart.
How do modern models encode position?: Most large models now use rotary embeddings, known as RoPE, which rotate the query and key vectors by an angle that depends on position. Relative distance then falls out of the maths naturally, and the scheme stretches to longer inputs better than the original fixed sine tables.
Is this why models get worse past their trained length?: Largely, yes. Beyond the positions seen in training, the encoding drifts into territory the model never learned to read. Most techniques for extending context work by rescaling or retraining those position signals rather than by changing attention itself.

Related explainers

Attention Embeddings Context window

One idea at a time, in your inbox

New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.

We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.

Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.