From vector back to a word
The block hands back a vector, not a word. How does that become the next token?
The idea inside
The unembedding scores every word in the vocabulary (its logits); softmax turns those scores into the odds you pick from.
After this lesson
You can explain how the final vector becomes the next word: the unembedding scores every word, softmax makes probabilities, then one is picked.
Where it leads
You've seen every part, and how it picks a word. Now find the one that's broken.
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
The unembedding scores every word in the vocabulary (its logits); softmax turns those scores into the odds you pick from.
The question it opens with
The block hands back a vector, not a word. How does that become the next token?
The walkthrough, in the lesson's own words
- So the vector has to be turned into scores. Watch how.
- The block hands back this vector. Predict how it becomes a word.
- Every word gets one score, its logit. The food words win.
- Softmax turns those raw scores into odds that add up to 100%.
- Pick the top word, add it, and the whole machine runs again.
- This vector is the model's meaning so far. How does it become a word?
- There is no lookup table. The vector is compared against every word the model knows, one score each.
- Each score is a logit: the final vector dotted with that word's row in the unembedding. It's the mirror of the embedding from 1.1, words became vectors, now a vector becomes scores. Those rows were learned in training too.
- Softmax squashes the raw logits into probabilities: every word gets a share of 100%, a bigger score gets a much bigger share. Now it's a set of odds.
- The loop is closed. A vector became a score for every word, then odds, then a pick. Add that word and the whole stack runs again for the next one.
- A model's vocabulary has ~100,000 words. What has to happen for every single token it writes?
- Every step, the final vector is scored against all ~100,000 words (the unembedding), softmax turns that into odds, one word is picked and appended, then the whole stack runs again. That per-word scoring is one more full matrix multiply, every single token.
- Do that for every word, and you have one score each. That's the unembedding.
Key takeaway
I can close the loop: a vector becomes a score for every word, then a choice.
What you can do after this lesson
You can explain how the final vector becomes the next word: the unembedding scores every word, softmax makes probabilities, then one is picked.
Check yourself: The transformer block outputs a vector. How does that become the next word?
- Score every word (logits), softmax into probabilities, then pick one(correct)
- Look the word up in a dictionary
- Return the vector to the user as-is
- Pick the word whose vector is longest
The final vector is scored against every word (the unembedding gives one logit each), softmax turns those logits into probabilities, and a word is picked. Then it all repeats.
Where it leads: You've seen every part, and how it picks a word. Now find the one that's broken.
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.