Why is verifying several tokens as cheap as generating one?

Because generation is limited by memory bandwidth. Producing one token means reading the whole model out of memory once, and the hardware has spare arithmetic capacity while it waits. Checking five candidate tokens in that same pass uses capacity that was idle anyway.

Where does the draft model come from?

Usually a much smaller model from the same family, so it shares the tokenizer and has similar habits. Some systems skip the second model and draft with extra heads on the main model or with a lookup over recent text.

Does it change the answer I get?

No, that is the point. Verification only accepts a draft token when it matches what the large model would have chosen, so speculative decoding is a pure latency optimisation rather than a quality trade.

Plain-language explainer

Speculative decoding, two models and one fast answer

What is speculative decoding and why does it make models faster?

A small fast model drafts the next few tokens, then the big model checks all of them in a single pass. Checking several tokens at once costs about as much as generating one, because the bottleneck is reading the weights out of memory, not the arithmetic. Every draft token the big model agrees with is free speed; the first disagreement is corrected and drafting restarts. The output is identical to what the big model would have produced alone. Only the wall clock changes.

Last reviewed July 25, 2026

Reading is the slow way. Start with a free lesson you can operate right now.

Start free: Predict the next word →

Free, no code, no signup.

Then go deeper: Speculative decoding: two models, one fast answer Locked

What people get wrong

It trades quality for speed. Rejected drafts are discarded, so the final text matches what the large model would have written by itself.
The small model needs to be almost as good. It only needs to be right often enough on easy tokens, which is most of them in ordinary prose.
It always helps. On unpredictable text the draft gets rejected constantly and the extra work can make things slower.

Where you see it in real products

The reason a hosted assistant streams smoothly at a price the provider can afford.
Local inference tools that pair a small and a large model on the same machine.
Latency-sensitive uses such as voice assistants and inline code completion.

Common questions

Why is verifying several tokens as cheap as generating one?: Because generation is limited by memory bandwidth. Producing one token means reading the whole model out of memory once, and the hardware has spare arithmetic capacity while it waits. Checking five candidate tokens in that same pass uses capacity that was idle anyway.
Where does the draft model come from?: Usually a much smaller model from the same family, so it shares the tokenizer and has similar habits. Some systems skip the second model and draft with extra heads on the main model or with a lookup over recent text.
Does it change the answer I get?: No, that is the point. Verification only accepts a draft token when it matches what the large model would have chosen, so speculative decoding is a pure latency optimisation rather than a quality trade.

Related explainers

KV cache Next-token prediction Inference economics

More in Speed, cost and control

Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.