Gradient descent: rolling downhill
Millions of knobs: how to tune them all?
The idea inside
Follow the slope of the loss downhill.
After this lesson
You can explain how training works: roll downhill, and why step size matters.
Where it leads
We tuned the knobs to predict better, but where did the meaning-map from Lesson 1.1 come from?
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
Follow the slope of the loss downhill.
The question it opens with
Millions of knobs: how to tune them all?
The walkthrough, in the lesson's own words
- Grab the ball. Drag it across the hill and watch the slope arrow flip.
- You're standing somewhere on the hill. Make a guess before we roll.
- Watch one run: each step moves downhill by rate × slope.
- Now you drive. Grab the ball to start anywhere, then drag the step size.
- A knob is one number inside the model. Turning cat's left-right coordinate from Act 1 is turning one knob; the model has millions.
- Millions of knobs, you can't check them all. But you can always feel which way is
- Last lesson, loss was your scoreboard, lower means less surprise. Picture it as height on a hill: a bad guess stands you up high, a good one drops you low. This curve is that height, drawn for one knob at a time.
- You're somewhere on the loss hill and you can't see the whole landscape. What can you actually figure out, right where you stand?
- Tempting, but there are millions of knobs, testing every combination would take longer than the universe has lasted. Too slow.
- Random pokes wander forever with no sense of better or worse. There's a smarter signal hiding under your feet.
- Exactly. You can measure the slope, how steeply the loss tilts at your spot, and step the other way. That direction is the gradient.
- The amber arrow points downhill, that's the slope, the gradient. The ball steps that way, the loss drops, and it slows as the hill flattens near the bottom.
- That's the whole of training: measure loss, find the slope, step downhill, repeat. Too big a step overshoots; too small crawls.
- Do this across millions of knobs at once, each on its own hill, all rolled in parallel, and the whole model gets better.
- Training a GPT-class model takes months on thousands of computers. What are they all doing that whole time?
- This same tiny move: measure the loss, feel the slope, nudge every knob, repeated trillions of times over the whole internet's text. The ball you rolled is one knob of millions being nudged at once. The step size still matters at that scale: too big and the run blows up, too small and months become years.
- We're watching just ONE of the millions of knobs.
- Every knob has its own hill, all rolled at once.
Key takeaway
Training is just downhill.
What you can do after this lesson
You can explain how training works: roll downhill, and why step size matters.
Check yourself: How does training actually improve the model?
- Follow the loss downhill (gradient descent)(correct)
- Add more example sentences by hand
- Increase the temperature
- Restart from scratch each step
Each step nudges the weights in the direction that lowers loss, a tiny step downhill. Repeat billions of times and the predictions get good.
Prefer reading first? This lesson has a companion explainer: Gradient descent
Where it leads: We tuned the knobs to predict better, but where did the meaning-map from Lesson 1.1 come from?
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.