How is distillation different from fine-tuning?

The source of the training data. Fine-tuning teaches a model from examples you collected. Distillation teaches a small model from a large model's answers, which is why you can generate as much training data as you can afford.

Can a student ever beat its teacher?

Not at what it was distilled on, though it can win on speed, cost and consistency. Students do sometimes score higher on narrow benchmarks when the distillation data was filtered to keep only correct answers, which removes some of the teacher's noise.

Is it legal to distil from a commercial model?

Provider terms usually forbid using their outputs to train a competing model, and this is an active area of dispute rather than a settled question. Check the terms of the model you are calling before building a product on its outputs.

Plain-language explainer

Distillation, how a small model learns from a big one

What is model distillation, and why is the mini version so good?

Distillation trains a small student model on the outputs of a large teacher model rather than on raw text alone. The teacher's answers are cleaner and more consistent than the internet, and in the richer versions the student also learns from how confident the teacher was across every option, not just its final pick. The student ends up far better than a model of its size trained from scratch. It inherits the teacher's style and its mistakes, and it does not exceed the teacher.

Last reviewed July 25, 2026

Reading is the slow way. Start with a free lesson you can operate right now.

Start free: From the internet to your answer →

Free, no code, no signup.

Then go deeper: Distillation: a small model learns from a big one Locked

What people get wrong

Distillation compresses the big model. It trains a separate small model to imitate it, which is why the student can behave differently in cases the teacher was never asked about.
It is the same as quantization. Quantization stores the same model with fewer bits per weight. Distillation trains a different, smaller model.
A distilled model is nearly as capable in general. It matches the teacher closely on the kinds of task it was distilled for and falls off outside them.

Where you see it in real products

The mini, flash and small tiers in every major model family.
On-device assistants in phones and laptops that never call a server.
Cheap high-volume classifiers built by distilling a frontier model onto one narrow task.

Common questions

How is distillation different from fine-tuning?: The source of the training data. Fine-tuning teaches a model from examples you collected. Distillation teaches a small model from a large model's answers, which is why you can generate as much training data as you can afford.
Can a student ever beat its teacher?: Not at what it was distilled on, though it can win on speed, cost and consistency. Students do sometimes score higher on narrow benchmarks when the distillation data was filtered to keep only correct answers, which removes some of the teacher's noise.
Is it legal to distil from a commercial model?: Provider terms usually forbid using their outputs to train a competing model, and this is an active area of dispute rather than a settled question. Check the terms of the model you are calling before building a product on its outputs.

Related explainers

Quantization Scaling laws Fine-tuning vs prompting

More in How models are trained

Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.