What does it mean when a model is called distilled?

It was trained, at least partly, to imitate a larger model's outputs instead of learning only from raw data. A distilled model is a separate, smaller network, not a compressed copy of the big one. Expect teacher-like behavior on the tasks it was distilled for and a thinner long tail everywhere else.

How is distillation different from fine-tuning?

The source of the training data. Fine-tuning teaches a model from examples you collected. Distillation teaches a small model from a large model's answers, which is why you can generate as much training data as you can afford.

What is a teacher-student model?

It is the standard name for the distillation setup. The teacher is a large trained model whose outputs, and sometimes whose full probability distributions, become the training signal. The student is the smaller model trained to match them. One teacher can generate data for a whole family of students at different sizes.

Can a student ever beat its teacher?

Not at what it was distilled on, though it can win on speed, cost and consistency. Students do sometimes score higher on narrow benchmarks when the distillation data was filtered to keep only correct answers, which removes some of the teacher's noise.

How much data does it take to distill a model?

Less than pretraining, more than a light fine-tune. A narrow single-task distill can work with a few thousand teacher outputs. Broad behavior takes much more: DeepSeek curated about 800,000 reasoning samples to create its R1 distills. Generating the data is a real cost, because every sample is a full run of the expensive teacher.

Is it legal to distil from a commercial model?

Provider terms usually forbid using their outputs to train a competing model, and this is an active area of dispute rather than a settled question. Check the terms of the model you are calling before building a product on its outputs.

Do reasoning models change how distillation works?

They changed what gets transferred. A reasoning model's visible chain of thought is part of its output, so the student trains on worked solutions instead of bare answers. That is why the small R1 distills got so good at math: they imitate the reasoning trace itself. The style transfers more reliably than the underlying judgment, so students still break on problems their teacher would catch.

Plain-language explainer

Distillation, how a small model learns from a big one

Model distillation is a training method where a small student model learns to imitate a large teacher model by training on the teacher's outputs, and sometimes its full probability distributions, instead of raw text alone. The result is a much cheaper model that keeps most of the teacher's skill on the tasks it was distilled for.

What is model distillation, and why is the mini version so good?

Distillation trains a small student model on the outputs of a large teacher model rather than on raw text alone. The teacher's answers are cleaner and more consistent than the internet, and in the richer versions the student also learns from how confident the teacher was across every option, not just its final pick. The student ends up far better than a model of its size trained from scratch. It inherits the teacher's style and its mistakes, and it does not exceed the teacher.

Last reviewed September 6, 2026

Reading is the slow way. Start with a free lesson you can operate right now.

Start free: From the internet to your answer →

Free, no code, no signup.

Ten seconds, no lesson: play with the standalone widget, and embed it in your own page.

Then go deeper: Distillation: a small model learns from a big one Locked

How does model distillation actually work?

Start with two models. The teacher is big, slow and capable. The student is small, fast and, at first, not capable at all. Distillation is any training recipe where the student learns from the teacher's behavior instead of only from raw text. Nothing is copied out of the teacher and it never shrinks. The student is a separate network that gets shaped, output by output, until it imitates the teacher closely on the tasks you care about.

The classic version, worked out for neural networks by Geoffrey Hinton and colleagues in 2015, trains the student on soft labels. When a language model predicts the next token, it does not just pick a winner. It assigns a probability to every token in its vocabulary. That full distribution is rich: it says that couch was nearly as plausible as sofa, and that banana never had a chance. Training the student to match the whole distribution, often softened with a temperature setting so the small preferences become visible, teaches it how the teacher weighs options, not just which one it chose.

The version that dominates the LLM era is plainer: synthetic-data distillation. You prompt the teacher at scale, keep its best outputs, and fine-tune the student on those input-output pairs like ordinary training text. No internal access is needed, which matters, since APIs rarely expose probabilities. DeepSeek built its R1 distills this way, curating roughly 800,000 reasoning samples from the full R1 model and fine-tuning much smaller Qwen and Llama models on them. The interactive distillation lesson on this site lets you run the teacher-student loop yourself at toy scale.

Why do labs distill their biggest models?

Because training happens once and serving happens forever. A frontier model is trained in one giant run, then answers millions of requests a day, and every answer costs compute. Most of those requests are routine. Serving every one with the biggest model means paying frontier prices, and frontier latency, for work a much smaller model could clear. Distilling the flagship into a small tier keeps most of the quality where the traffic actually lives, at a fraction of the cost per request.

There is a second, quieter reason: teacher outputs are better training data than the internet. The raw web is contradictory, messy and full of junk. A strong teacher's answers are consistent, well formatted and already shaped like the behavior you want. A small model trained on them ends up far ahead of an identical model trained from scratch on scraped text.

The pattern is everywhere once you look. Google described the first Gemini Flash as distilled from its larger Pro sibling, and its open Gemma family credits knowledge distillation in its technical reports. Meta said its phone-sized Llama 3.2 models were built by pruning bigger Llamas, then recovering quality with distillation that used the larger models' output probabilities. And every major lineup ships mini, flash or small tiers priced far below the flagship. Vendors rarely detail the recipe, but distillation is the standard tool for that job.

Distillation vs quantization vs pruning: what is the difference?

All three make models cheaper to run, and people mix them up constantly. Distillation trains a new, smaller model to imitate a big one. Quantization keeps the same model but stores each weight in fewer bits, trading a little precision for a lot of memory. Pruning deletes weights, neurons or whole layers from an existing model, usually followed by extra training to heal the damage.

The practical difference: a quantized model is the same brain slightly blurred, so it behaves almost identically. A distilled model is a different brain educated by the first one, so it can behave differently on anything it was never taught. Pruning sits in between, the same brain with pieces removed. They also stack. Llama 3.2's small models were pruned and then distilled, and phone deployments often quantize the distilled student on top, so one on-device model can be all three at once.

Three ways to shrink the cost of a model. They answer different questions and are often combined.

Technique	What changes	Retraining needed	Typical use
Distillation	A new smaller model learns to imitate the big one	Yes, a full training run for the student	Mini and flash tiers, on-device assistants
Quantization	Same model, fewer bits per weight, like 16-bit down to 4-bit	Little to none, usually just calibration	Running open models on consumer hardware
Pruning	Weights or whole layers are deleted	Usually, to recover the lost accuracy	Trimming a model, often paired with distillation

How much smaller and cheaper is a distilled model?

There is no fixed ratio, but the published examples cluster in a recognizable range: dramatic savings in size and speed, modest losses on the measured tasks. The cleanest early number comes from DistilBERT in 2019: 40 percent fewer parameters and 60 percent faster, while keeping about 97 percent of the teacher's measured language understanding. That shape, most of the quality for a fraction of the cost, is the whole sales pitch of distillation.

The modern showcase is DeepSeek-R1. The full teacher is a 671-billion-parameter mixture-of-experts model, hardware few people can run. Its distilled students, released alongside it in early 2025, run from 70 billion parameters down to 1.5 billion. Per DeepSeek's own evaluations, the 32B distill beat OpenAI's o1-mini on reasoning benchmarks with roughly one twentieth of the teacher's total parameters, and even the laptop-sized 1.5B version beat far larger general models on competition math while staying weak outside that lane.

Price follows size. Within a provider's lineup as of mid-2026, the small tier typically costs roughly 5 to 10 times less per token than the frontier tier, and it answers faster. Treat retained-quality numbers with care, though: they are measured on the tasks the student was distilled for, and the honest expectation outside them is lower.

Reported examples, rounded. Retained quality is task-dependent: high on what the student was distilled for, lower outside it.

Student vs teacher	Size and speed change	What was kept
DistilBERT vs BERT, 2019	40% fewer parameters, 60% faster	About 97% of measured language understanding
R1-Distill-Qwen-32B vs DeepSeek-R1, 671B	Roughly 20x fewer total parameters	Beat o1-mini on reasoning benchmarks, per DeepSeek
R1-Distill-Qwen-1.5B vs DeepSeek-R1, 671B	Hundreds of times smaller, laptop-sized	Competition math stayed strong, broad ability did not
Small API tier vs frontier tier, mid-2026	Roughly 5 to 10x cheaper per token	Most routine requests, with the hard tail falling off

When does distillation lose?

The ceiling is the teacher. On the tasks it was distilled for, a student can approach the teacher and, when the training data was filtered to keep only the teacher's correct answers, occasionally edge past its average. It does not surpass what the teacher knows. And if the teacher hallucinates a fact, prefers a wrong pattern or carries a bias, the student learns that too, at industrial scale.

The sharper failure is narrowness. Capacity is exactly what you removed, and the first things a small model drops are rare knowledge and unusual combinations. A distill tuned on reasoning traces can ace math contests and still fumble general questions its teacher answers easily. A well-known 2023 study of imitation models found that students pick up the teacher's confident style more reliably than its factual accuracy, which makes them better at sounding right than at being right.

There is also an evaluation trap. A distilled model tends to look excellent on benchmarks close to its training data, because that is precisely what it was optimized toward. The gap shows up on your real traffic, in the odd requests no benchmark covered. Before swapping a frontier model for a distill, run your own evals on your own examples, especially the ugly ones.

Can you distill from a model behind an API, and is it allowed?

Technically, yes, and that is exactly why the synthetic-data recipe took over. All you need is outputs, and outputs are what an API sells. Most commercial APIs will not give you the full probability distributions, so the classic soft-label recipe is out of reach, but plain question-answer pairs are enough to move a lot of capability.

Whether you may is a different question. The terms of service of major closed providers, including OpenAI, Anthropic, Mistral and xAI, prohibit using their outputs to build competing models. The rule made front pages in early 2025, when OpenAI said it had evidence that DeepSeek had trained on its model outputs, after earlier blocking accounts it suspected of harvesting data at scale. The dispute stayed contested rather than settled, and it turned distillation from a training-lab term into a geopolitical one.

There are fully sanctioned lanes too. Open-weight licenses generally allow distilling the models you download, sometimes with attribution rules, and DeepSeek released its own distills openly. Providers even sell the workflow inside their own families: OpenAI offers a distillation pipeline in its API platform that stores a big model's outputs and fine-tunes a smaller one on them, and Amazon Bedrock sells managed distillation the same way. Distilling the expensive tier you already use into the cheap one is a supported product path. Across vendor lines, read the terms first. None of this is legal advice.

What people get wrong

Distillation compresses the big model. It trains a separate small model to imitate it, which is why the student can behave differently in cases the teacher was never asked about.
It is the same as quantization. Quantization stores the same model with fewer bits per weight. Distillation trains a different, smaller model.
A distilled model is nearly as capable in general. It matches the teacher closely on the kinds of task it was distilled for and falls off outside them.
Distillation is theft by definition. Labs distill their own models constantly and sell tooling for it. The disputes are about distilling someone else's model against its terms of service, not about the technique.

Where you see it in real products

The mini, flash and small tiers in every major model family.
On-device assistants in phones and laptops that never call a server.
Cheap high-volume classifiers built by distilling a frontier model onto one narrow task.
Open-weight reasoning distills, like the R1 versions of Qwen and Llama, small enough to run on one GPU or a laptop.

Common questions

What does it mean when a model is called distilled?: It was trained, at least partly, to imitate a larger model's outputs instead of learning only from raw data. A distilled model is a separate, smaller network, not a compressed copy of the big one. Expect teacher-like behavior on the tasks it was distilled for and a thinner long tail everywhere else.
How is distillation different from fine-tuning?: The source of the training data. Fine-tuning teaches a model from examples you collected. Distillation teaches a small model from a large model's answers, which is why you can generate as much training data as you can afford.
What is a teacher-student model?: It is the standard name for the distillation setup. The teacher is a large trained model whose outputs, and sometimes whose full probability distributions, become the training signal. The student is the smaller model trained to match them. One teacher can generate data for a whole family of students at different sizes.
Can a student ever beat its teacher?: Not at what it was distilled on, though it can win on speed, cost and consistency. Students do sometimes score higher on narrow benchmarks when the distillation data was filtered to keep only correct answers, which removes some of the teacher's noise.
How much data does it take to distill a model?: Less than pretraining, more than a light fine-tune. A narrow single-task distill can work with a few thousand teacher outputs. Broad behavior takes much more: DeepSeek curated about 800,000 reasoning samples to create its R1 distills. Generating the data is a real cost, because every sample is a full run of the expensive teacher.
Is it legal to distil from a commercial model?: Provider terms usually forbid using their outputs to train a competing model, and this is an active area of dispute rather than a settled question. Check the terms of the model you are calling before building a product on its outputs.
Do reasoning models change how distillation works?: They changed what gets transferred. A reasoning model's visible chain of thought is part of its output, so the student trains on worked solutions instead of bare answers. That is why the small R1 distills got so good at math: they imitate the reasoning trace itself. The style transfers more reliably than the underlying judgment, so students still break on problems their teacher would catch.

Related explainers

Quantization Scaling laws Fine-tuning vs prompting

One idea at a time, in your inbox

New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.

We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.

Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.