Where Query, Key and Value come from

What this lesson shows

Q/K/V are the embedding times three learned weight grids (W_Q, W_K, W_V); training tunes the grids, nobody writes them.

The question it opens with

One word has one embedding, so where do three different vectors (Query, Key, Value) come from?

The walkthrough, in the lesson's own words

• One word has one embedding. So where do three different vectors come from?
• Multiply the embedding by one grid. Out comes the Query.
• Tap Query, Key, or Value. Same embedding, a different grid each time.
• “drinks” needs three versions of itself: one to ask, one to answer, one to hand over. One embedding goes in. How are the three made?
• Right. Nothing is stored per word. One embedding is multiplied by three different grids, and each grid reshapes it into a different view. Watch one grid do it.
• Copying would give three identical vectors, useless. Instead each of three grids multiplies the embedding into a different view. Watch one grid do it.
• Nothing is saved per word. The three views are computed on the spot: the embedding times three different grids. Watch one grid do it.
• Each output number is one grid row times the embedding, added up, the same dot product from 1.3. One grid in, one whole vector out.
• Toy numbers, three cells wide. A real embedding is hundreds of numbers and the grids are learned, but the multiply-and-add is exactly this.
• One row turns the whole embedding into a single number. Stack a row per output cell and you rebuild a full vector, that stack of rows is the grid (a weight matrix). You meet the same move again in 3.6 as the layer after attention.
• Nobody fills in these grids by hand. They start as random numbers, and training (2.4) nudges them again and again until Queries and Keys that should match line up, because that made the next word easier to guess (Act 2).
• Engineers program the grids so that verbs go looking for their subject.
• No one writes those numbers. The grids start random and gradient descent tunes them until matches like verb-to-subject help predict the next word. The behavior is discovered in training, not coded.
• Because a trained W_Q made “drinks”’s Query and a trained W_K made “cat”’s Key point almost the same way, so their dot product landed high. Change the training and those grids change, and so does every score. The 2.7 was never set by hand.
• People say fine-tuning a model on your data “teaches it your style”. In terms of what you just saw, what literally changes inside the model?
• The numbers in grids like these. Fine-tuning nudges the same weight matrices a bit further, so the Queries, Keys, and Values come out slightly different and the model leans on different words. Nothing new is stored; the existing grids are re-tuned.
• Three jobs, so “drinks” needs three versions of itself. One embedding, three vectors.
• Trained grids: “drinks” now reaches “cat”, its subject.
• Random grids: the reach is weak and points nowhere useful.
• A picture of the effect, not a real training run.

Key takeaway

Three learned grids reshape one embedding into three views, and training set their numbers.

What you can do after this lesson

You can explain that Query, Key and Value are made by multiplying the embedding by three weight matrices that training learns.

Where it leads: Now turn those matches into an updated meaning.

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.