Distillation: a small model learns from a big one
Frontier models are huge and pricey to serve. Must every task pay the full cost?
The idea inside
A small student model imitates a big teacher, far cheaper to run.
After this lesson
You can explain distillation: a small student model trained to mimic a big teacher, for cheap inference.
Where it leads
Big or small, its weights are fixed after training, so can you still teach it anything new?
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
A small student model imitates a big teacher, far cheaper to run.
The question it opens with
Frontier models are huge and pricey to serve. Must every task pay the full cost?
The walkthrough, in the lesson's own words
- Drag the student smaller. How much of the teacher's quality survives the shrink?
- Predict: how could a small model get nearly as good as a giant one?
- Most of what you ask is routine. Must every task pay frontier prices?
- A small student inherits most of a big teacher's skill, cheap, fast, nearly as good.
- Smaller student = cheaper and faster to run, until quality finally cracks.
- Smaller, faster, and still nearly the teacher.
- Shrunk too far: the copied skill finally starts to slip.
- How could a small model end up nearly as good as the giant?
- Trained from scratch, a small model lands well below the giant, but the giant already knows the answers.
- Not quite, more raw text, or fewer bits per weight (that's quantization), is a different trick. The richer move: learn from the giant itself.
- The small student trains on the teacher's answers and probabilities, not raw text, a far richer signal than learning from scratch.
- The teacher does not just say "positive", it says 80% positive, 15% mixed. Every example carries the whole ranking, not just one right answer.
- Most of what you ask it, summaries, translations, quick classifications, is routine. Paying frontier prices for every one is wasteful.
- Both make models cheaper, but differently. Quantization (lesson 4.6) keeps the same model and stores each weight in fewer bits. Distillation builds a genuinely smaller model that learned to mimic a bigger one. You can even do both: distill, then quantize.
- A company ships a small, cheap model that feels almost as sharp as the frontier giant it sits next to. How did such a small model get so good?
- Distillation: the small student was trained to imitate the big teacher's answers, inheriting most of the skill at a fraction of the size and cost. That is why most of the fast, cheap models you use day to day can punch well above their size.
- The student trains on the teacher's answers, copying what to say, not just raw text.
- Illustrative fidelity numbers, real distillation results vary by task.
- Look at the gap: the student bar drops a lot on size, barely on quality.
Key takeaway
A small student can inherit most of a big teacher's skill, cheap to run, nearly as good.
What you can do after this lesson
You can explain distillation: a small student model trained to mimic a big teacher, for cheap inference.
Check yourself: What is model distillation?
- Training a small student model to imitate a big teacher(correct)
- Removing bits from each weight
- Deleting rare tokens from the vocabulary
- Caching past outputs
Distillation trains a small student model to copy a big teacher's outputs, packing much of the quality into a cheaper, faster model.
Prefer reading first? This lesson has a companion explainer: Distillation
Where it leads: Big or small, its weights are fixed after training, so can you still teach it anything new?
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.