The transformer block, assembled & stacked
How do the parts fit into the whole machine?
The idea inside
attention + FFN + residual/norm = a block; stack N blocks (nano-GPT).
After this lesson
You can describe a whole transformer block and how stacking it predicts the next word.
Where it leads
But the stack hands back a vector, not a word. How does that become the next token?
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
attention + FFN + residual/norm = a block; stack N blocks (nano-GPT).
The question it opens with
How do the parts fit into the whole machine?
The walkthrough, in the lesson's own words
- Tap any part to inspect what it does. Each one you already met.
- The cat drinks ___ goes in. Tap the output slot to predict what comes out.
- You can see every part, and you know what each one does.
- That's the whole engine behind ChatGPT and Claude.
- One last step turns that final vector into the table above: a projection called the unembedding gives every vocabulary word a score, and softmax turns the scores into probabilities. “milk” just scores highest here, but picking from this table is the next lesson.
- Four parts you already knew, in one block, stacked 12 to 100+ times. That stack is the whole engine, and every number inside it was learned by training (Act 2), not written by a person.
- When ChatGPT or Claude answers you, what is the model actually doing under the hood, the same thing every time?
- It runs your words up through this transformer block, stacked dozens of times, and the stack hands back one score per vocabulary word. Everything it can do, code, poems, answers, comes from this same repeated block predicting the next word.
Key takeaway
I can see the whole machine, and I know every part.
What you can do after this lesson
You can describe a whole transformer block and how stacking it predicts the next word.
Check yourself: What is a transformer 'block', and how do you build a big model from it?
- Attention plus a small per-word network plus residual/norm, stacked many times(correct)
- A single giant lookup table
- One block per training example
- A block of memory for storing answers
One block is attention plus a small per-word network plus stabilizing tricks (residuals, layer norm). Stack the same block many times and you get the engine behind ChatGPT and Claude.
Prefer reading first? This lesson has a companion explainer: Transformers
Where it leads: But the stack hands back a vector, not a word. How does that become the next token?
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.