Distillation: shrink the student, keep the teacher's skill

1. 1Predict: how could a small model end up nearly as good as the giant? Pick your answer before the reveal.
2. 2Watch the student train on the teacher's answers and probabilities, not on raw text.
3. 3Drag the student smaller. Quality holds, and holds, and then finally cracks.

Most of what people ask a model is routine: summaries, translations, quick classifications. Paying frontier prices for every one is wasteful, so labs build small models. The question this widget opens on is how a small model can get close to a giant one, and most people guess wrong. More raw text does not do it, and fewer bits per weight is a different trick called quantization.

The richer move is to learn from the giant itself. The teacher does not just say positive, it says 80 percent positive, 15 percent mixed. Every example carries the teacher's whole ranking, not just one right answer, and that is a far richer training signal than scraped text. The student, a separate and much smaller network, trains on those outputs until it imitates the teacher closely.

Then you take the size slider and pull the student down. The size bar drops a lot and the quality bar barely moves, until you go too far and the copied skill starts to slip. That gap between the two bars is the entire business case for the mini and flash tiers in every model family.

Distillation is how the fast, cheap models you use day to day punch above their size. DeepSeek built its R1 distills by fine-tuning much smaller models on hundreds of thousands of the full model's reasoning samples. Google described Gemini Flash as distilled from Pro. The student inherits the teacher's style and its mistakes, and it does not exceed the teacher on the tasks it was distilled for, which is the honest limit you will feel when the quality bar cracks.

Paste this into any HTML page, course platform, wiki or slide tool that accepts an iframe. It runs with no account, sets no cookies of its own, and links back here.

Attribution is built in. If you write about it, a link to this page is all we ask.

Is distillation the same as quantization?

No. Quantization keeps the same model and stores each weight in fewer bits. Distillation trains a genuinely smaller model that learned to mimic a bigger one. You can do both: distill first, then quantize the student for a phone.

Can a distilled student beat its teacher?

Not at what it was distilled on, though it wins on speed, cost and consistency. Students sometimes score higher on narrow benchmarks when the training data kept only the teacher's correct answers, which strips out some of the teacher's noise.

Can I use this widget in a course or a talk?

Yes. Copy the iframe snippet on this page into any HTML page or slide tool. It runs without an account and links back to the full distillation lesson and explainer.