Speculative decoding: two models, one fast answer

What this lesson shows

A small model drafts tokens ahead; the big one verifies them at once.

The question it opens with

The model writes one token at a time, in order. That's slow, can we cheat the wait?

The walkthrough, in the lesson's own words

• Drag the draft window and the hit rate. Watch the slow steps drop.
• Predict: drag the hit rate down to where you think the speedup dies.
• The order is fixed, yet checking beats writing. Here's why.
• Draft cheap, verify once, keep what you agree with. Same answer, faster.
• Each row drafts ahead, the big model verifies once, only the ✓ prefix survives.
• Fewer ✕ means fewer slow passes, that's the whole win.
• This trick runs invisibly under most chats you have: it is part of why replies stream faster and cost less than a year ago, same model, same words.
• Slide it down: how low can the drafter go before there's no speedup left?
• Still a win. As long as the drafter lands more than one token most rounds, you skip slow passes, the output is identical either way.
• A bad drafter gets most guesses rejected, so almost every round advances by one token, back to one slow pass each. The draft work is wasted.
• Most words are easy. After "once upon a", even a tiny model knows "time". The big model is only really needed at the hard forks, that is why the drafter's hit rate is high.
• The order is fixed, token 7 can't be written before token 6.
• Writing is slow because each token has to wait for the one before it, token 7 can't start until token 6 exists. But the drafter already wrote the guessed tokens down. So instead of writing, the big model only has to check each position, "is this the token I'd have picked here?", and checking needs no waiting, because every token it's checking already exists. That lets it run the whole window through together in one pass, while writing from scratch needs one slow waiting-step per token.
• Draft cheap, verify once, keep the prefix you agree with.
• In 2026 serving, the drafter often isn't a separate model: the big model grows tiny extra drafting heads that guess a few tokens ahead for it (EAGLE, Medusa), then it verifies its own draft the same way.
• Your chatbot's replies stream noticeably faster than last year, same model name, same quality. What likely changed under the hood?
• Speculative decoding (draft + verify) and tricks like it: a small drafter races ahead, the big model checks in bulk, identical words, fewer big-model passes.
• Each chip marked V is one slow big-model pass, it verifies the whole drafted window at once and keeps the ✓ prefix, discarding any ✕ after the first miss.
• Even a fully rejected round still moves forward one token: the verify pass computes the right next token anyway, a free correction.

Key takeaway

A draft-and-verify trick speeds up generation with no loss in quality.

What you can do after this lesson

You can explain speculative decoding: a small model drafts, the big model verifies, for faster output.

Prefer reading first? This lesson has a companion explainer: Speculative decoding

Where it leads: Another way to go faster and cheaper: shrink the model itself.

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.