Scaling laws
Does throwing more at it predictably help?
The idea inside
Loss falls as a power law in params, data, compute (Chinchilla).
After this lesson
You can explain scaling laws: loss falls predictably with compute, but predicts loss not skills.
Where it leads
How to add parameters without paying for them every token?
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
Loss falls as a power law in params, data, compute (Chinchilla).
The question it opens with
Does throwing more at it predictably help?
The walkthrough, in the lesson's own words
- Back to the wheel. Push effort as far as it goes.
- Invented curve with the shape of the published trend; the model dots are placed on it, not measured.
- More effort, less wrong, by a steady fraction.
- Drag the effort slider up and watch how-wrong fall.
- Loss is the model's surprise at the truth (from 2.3), lower is better. Training effort means scaling the three ingredients together: model size, training data, and compute.
- Feel that? It isn't chaos and it isn't a wall. Each 10× of effort closes the same predictable fraction of the remaining gap to the floor, a smooth, repeatable curve.
- Down and to the right: more effort, less wrong. Real models are marked for scale on an invented curve shaped like the published trend. Next, take the wheel.
- Scaling reliably lifts average capability, but at higher cost and latency and with diminishing returns, and a small model can still win on a specific, easy task.
- A lab announces a new model trained on far more data and compute, and reviewers expect it to be steadily better. Why is that a reasonable bet, and where does the bet get shaky?
- Scaling laws say loss falls along a smooth, predictable curve as you scale data, parameters, and compute together, so a bigger, better-fed model is a safe bet to lower loss. The shaky part is which specific abilities appear and when, so treat broad capability gains as likely but test each concrete skill yourself rather than assuming it.
Key takeaway
Bigger + more data predictably lowers loss, but predicts loss, not capabilities.
What you can do after this lesson
You can explain scaling laws: loss falls predictably with compute, but predicts loss not skills.
Check yourself: What do scaling laws predict?
- Loss falls predictably with compute (not which skills appear)(correct)
- Exactly which abilities emerge
- The best learning rate
- The ideal vocabulary size
Scaling laws forecast lower loss from more compute and data, smoothly. They don't tell you which specific abilities will show up.
Prefer reading first? This lesson has a companion explainer: Scaling laws
Where it leads: How to add parameters without paying for them every token?
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.