Plain-language explainer
Positional encoding, explained
How does a transformer know word order, if attention looks everywhere at once?
Attention on its own treats a sentence as a bag of words: 'dog bites man' and 'man bites dog' would look identical. So models mix a position signal into every token's vector before attention runs. Each position gets its own mathematical stamp, and the stamp becomes part of the token's meaning, so 'first word' and 'seventh word' are different even for the same word. Many modern models use a rotating variant, RoPE, which encodes relative distance between words.
Do not just read it. Operate the mechanism yourself in a short interactive lesson.
See it work: How the model knows word order →Free, no code, no signup.
What people get wrong
- Models read left to right the way people do. Attention sees every token at once. Order exists only because it was stamped in.
- Word order is stored in the words themselves. The same word at two positions gets two different vectors, and only the stamp differs.
- Position is a minor implementation detail. Stretching models to longer contexts is hard largely because the position scheme must stretch with them.
Where you see it in real products
- Translation tools keep who-did-what-to-whom straight because position is part of meaning.
- Code assistants track nesting and bracket order through the same signal.
- Long-context model releases often name their position-scheme tricks, RoPE scaling among them, in the fine print.
Related explainers
Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.