Keeping the signal alive
Deep stacks forget the original signal or blow up.
The idea inside
Residual = keep a copy and add the change; LayerNorm = rescale to stay sane.
After this lesson
You can explain residuals and LayerNorm: the tricks that make deep nets trainable.
Where it leads
Assemble the parts into one block, then stack them.
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
Residual = keep a copy and add the change; LayerNorm = rescale to stay sane.
The question it opens with
Deep stacks forget the original signal or blow up.
The walkthrough, in the lesson's own words
- Tap the top of the lane: what reaches the output after every layer rewrites it?
- Skip path off: the signal must pass through every layer. Watch it fade.
- Flip the skip path on. Now each layer adds to a copy of the original.
- Adding grows the signal. LayerNorm resets it to a sane size.
- Each layer adds to the signal, then LayerNorm keeps it in shape.
- Right. That's the fade you're watching: each reshape buries the original more, until the top has no idea what came in. Next: the fix.
- Not quite. With nothing protecting it, the original smears into noise: each reshape buries it more, until the top has no idea what came in. Next: the fix.
- LayerNorm acts on the copy a block reads, not on the residual stream after the add. So the stream itself is left untouched and just keeps accumulating, while every block still sees inputs of a similar size.
- The skip path is a shortcut that lets the signal skip past a layer's work.
- Nothing gets skipped. The layer still runs; its change is added on top of an untouched copy of the signal, so the original is never lost.
- Years ago, networks stopped improving past a certain depth, accuracy got worse the more layers you added. Today’s models stack 100+ layers and keep getting better. What changed?
- Residual skip paths plus LayerNorm: each layer adds to an untouched copy of the signal instead of overwriting it, and LayerNorm keeps the size sane. That is the trick that made very deep stacks trainable, and it is in every transformer block you use.
- A vertical signal lane with three tappable outcome swatches at the top: predict what reaches the output after every layer rewrites the signal.
- A vertical signal lane with the skip path on and LayerNorm on: the stream stays clean while each block reads a rescaled, sane-sized copy.
- A vertical signal lane with the skip path on but LayerNorm off: the copy each block reads balloons too big.
- A vertical signal lane with the skip path off: the signal fades to mush near the top.
Key takeaway
Shortcuts keep deep nets trainable.
What you can do after this lesson
You can explain residuals and LayerNorm: the tricks that make deep nets trainable.
Check yourself: Why do transformers add a block's input back onto its output (a residual)?
- So the original signal survives through a deep stack of layers(correct)
- To make the model smaller
- To translate between languages
- To pick the next word directly
Residual connections add each block's input back to its output, so the original signal survives through a deep stack and training stays stable. Without them, deep models are hard to train.
Where it leads: Assemble the parts into one block, then stack them.
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.