How the model knows word order
Attention is order-blind: 'cat drinks milk' = 'milk drinks cat'.
The idea inside
Inject position (sinusoidal / RoPE intuition).
After this lesson
You can explain why models add positional information so word order matters.
Where it leads
Does bigger always mean better?
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
Inject position (sinusoidal / RoPE intuition).
The question it opens with
Attention is order-blind: 'cat drinks milk' = 'milk drinks cat'.
The walkthrough, in the lesson's own words
- Add position info, then watch the meaning lock in.
- Take a guess: how could the model tell the order apart?
- Order locked: each word now carries its slot, so attention's Query·Key scores (from 3.3) differ by position, this can only mean
- Because each knob turns smoothly, two nearby slots leave it in almost the same spot, so close positions get near-identical stamps, and on average the gap grows with distance (it can wobble a little for far-apart slots).
- Same glyphs, new slots, and now each one carries its slot's marker, so the order genuinely reads differently.
- Same glyphs, new slots, but with no markers, attention sees the same order-blind soup. Now add position info.
- Swap the two nouns: the same tiles will trade places. Try it with markers off, then on.
- The words are all jumbled together, order-blind. What's the simplest fix that lets the model tell “the cat drinks milk” from “milk drinks cat”?
- Right, give each slot its own marker, add it to the word, and now “first cat” and “third cat” are different. Cheap, and it works for any sentence.
- Not quite, there's no “correct” order to sort back to; the order IS the meaning. The model needs the order baked into each word, not undone.
- Not quite, memorizing can't scale to every possible sentence. The trick is far cheaper: tag each slot with its own marker, then add it to the word.
- Each token's fingerprint is read off the ruler's volume knobs, set at different speeds, so “first” and “third” land in different spots. Add that fingerprint to the word and attention's Query·Key scores now depend on position, order survives the dot product.
- You paste a document far longer than anything the model trained on, and answers about early vs. late sections get shaky. Why can “what came first” blur at extreme lengths?
- Attention scores pairs with a dot product, which is order-blind on its own, so each token is stamped with a position fingerprint before attention runs. Stretch the input far past the lengths the model trained on and those stamps land in territory it never practiced, so the position signal gets weak and “what came first” can blur.
Key takeaway
Position is added, not assumed.
What you can do after this lesson
You can explain why models add positional information so word order matters.
Prefer reading first? This lesson has a companion explainer: Positional encoding
Where it leads: Does bigger always mean better?
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.