Query, Key, Value
How does a token decide what to look at?
The idea inside
Each token emits a Query, a Key, and a Value; match Q to K by dot product.
After this lesson
You can explain Query, Key, Value and that attention reuses the dot product.
Where it leads
Those matches used a Query and a Key. But where do those come from?
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
Each token emits a Query, a Key, and a Value; match Q to K by dot product.
The question it opens with
How does a token decide what to look at?
The walkthrough, in the lesson's own words
- “drinks” holds up its Query. Tap the word whose folder tab (its Key) you think matches best.
- Attention reuses the dot product you already know.
- Right, “cat” is the subject doing the drinking. Watch the match play out.
- Each word starts as its single embedding (from 1.2). The model multiplies that embedding by three separate grids of numbers to make three views, a Query, a Key, and a Value. Each view is a vector, so the score is just their dot product, the same one from Lesson 1.3.
- Those grids are learned in training, nobody writes them by hand. The next lesson builds Query, Key and Value step by step and shows exactly where the numbers come from.
- Attention must run on some brand-new kind of math built just for language.
- It reuses the dot product from 1.3. A Query dotted against each Key gives the scores, and the top score wins the biggest share of Value.
- A word’s Value is the information it carries, roughly, its meaning. Because cat wins the biggest share, most of cat’s meaning flows into drinks, so drinks now knows a cat is its subject.
- In the sentence “I left the bank to fish by the river,” a model reads “bank” as a riverbank, not a place for money. What machinery decides which meaning to pull in?
- “bank” sends a Query, “river” answers with a Key that scores high (Q · K), so “river”’s Value flows in and shifts “bank” toward the water sense. Disambiguating words by context is just Query, Key, and Value matching, the same dot-product search you saw here.
- Where Q, K and V come from: one embedding, reshaped into three views.
- Those 2.7 and 0.4 are the same scores you saw on the bars. Multiply each matching slot, add them up, and that is the whole score.
- Readable toy numbers. The real Query and Key numbers are learned in training, the next lesson shows where they come from.
Key takeaway
Attention reuses the similarity trick I already learned.
What you can do after this lesson
You can explain Query, Key, Value and that attention reuses the dot product.
Check yourself: In attention, how does a token decide which others to focus on?
- It matches its Query against every Key with a dot product(correct)
- It always picks the nearest tokens in the sentence
- It focuses on the most frequent words
- It attends to every token equally
Each token emits a Query and a Key. Attention scores a Query against every Key with the dot product, the same similarity trick from 1.3, and focuses where they align.
Prefer reading first? This lesson has a companion explainer: Query, Key, Value
Where it leads: Those matches used a Query and a Key. But where do those come from?
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.