LABS · 16  /  INFERENCE

The Understudy.

A giant model reads out one token at a time — slow, because each token drags all its weights through the GPU. So hire an understudy: a small fast model guesses the next few tokens, and the star checks them all in one pass, keeping every guess it agrees with. Same performance, a fraction of the passes.

Act 1 · Why decode is slow
One token, one whole model
Generating text token-by-token means one forward pass per token — and each pass streams every weight of a 70B model through the GPU. The compute is trivial; the memory movement is the wall. Watch the passes stack up.
Output (1 forward pass each)
0
Tokens produced
0
Forward passes
1.0
Tokens / pass
Standard autoregressive decode: tokens-per-pass is stuck at exactly 1. Every token pays the full memory cost of the model. Speculative decoding attacks precisely this ratio.
Act 2 · The trick
Draft, then verify in one pass
The draft model (small, fast) guesses the next 4 tokens. The target model checks all 4 in a single pass, accepting each while it agrees — until the first mismatch, which it corrects. Then the draft guesses again from there.
Draft model proposes
Target verifies → committed output
0
Tokens committed
0
Target passes
Tokens / pass
Every green token was produced "for free" — verified in a pass that would otherwise make just one token. The one yellow correction per round is the target having the final say, which is what keeps output identical.
Act 3 · The dial that matters
Acceptance rate is everything
Speedup rides entirely on how often the draft agrees with the target. A draft that mimics the target well gets long accepted runs; a poor one gets rejected constantly. Tune the acceptance rate and the draft length, and watch the speedup respond.
Draft acceptance rate70%
Tokens drafted per round (K)4
Avg accepted / round
Speedup vs plain
Optimal K here
There's a sweet spot in K: draft too few and you underuse the free parallel pass; draft too many and you waste the small model on tokens that get rejected anyway. The best K rises as acceptance rises.

Why guessing is faster than working

The counterintuitive core: the big model still runs, yet you finish sooner. The resolution is that LLM decode is memory-bound. Producing one token means streaming every weight of the model through the GPU once; the actual arithmetic barely registers next to that memory traffic. So a forward pass has enormous spare compute — you're paying to move the weights whether you use that compute or not.

Speculative decoding spends that spare compute. Instead of asking the target "what's the next token?", you ask "here are 4 proposed tokens — for each position, what would you have said?" — and it answers all 4 in one weight-streaming pass. Every proposal it agrees with is a token you got without paying a second pass.

The lossless guarantee

It's not a quality gamble. The acceptance test uses the target's own probabilities, and rejected positions are re-sampled from the target — so the output is provably distributed exactly as if the target generated every token. You are not trading accuracy for speed. You're trading spare compute for it.

Where the draft comes from

The draft model just needs to be fast and usually right: a small model from the same family, a distilled version, or even the model's own extra prediction head (as in Medusa or the multi-token-prediction head DeepSeek-V3 trains). The closer the draft mimics the target, the higher the acceptance rate, the bigger the win.

The tuning

Two knobs. Acceptance rate is set by draft quality — better draft, longer accepted runs. K, the number of tokens drafted per round, has a sweet spot: too small underuses the free parallel verification, too large wastes draft effort on tokens likely to be rejected. Real systems see roughly 2-3× faster generation — pure latency, identical output.

Check yourself

Four questions. No guessing — well, a little.

Finished this one? 0 / 22 Labs done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Labs