A giant model reads out one token at a time — slow, because each token drags all its weights through the GPU. So hire an understudy: a small fast model guesses the next few tokens, and the star checks them all in one pass, keeping every guess it agrees with. Same performance, a fraction of the passes.
Act 1 · Why decode is slow
One token, one whole model
Generating text token-by-token means one forward pass per token — and each pass streams every weight of a 70B model through the GPU. The compute is trivial; the memory movement is the wall. Watch the passes stack up.
Output (1 forward pass each)
0
Tokens produced
0
Forward passes
1.0
Tokens / pass
Standard autoregressive decode: tokens-per-pass is stuck at exactly 1. Every token pays the full memory cost of the model. Speculative decoding attacks precisely this ratio.
Act 2 · The trick
Draft, then verify in one pass
The draft model (small, fast) guesses the next 4 tokens. The target model checks all 4 in a single pass, accepting each while it agrees — until the first mismatch, which it corrects. Then the draft guesses again from there.
Draft model proposes
Target verifies → committed output
0
Tokens committed
0
Target passes
—
Tokens / pass
Every green token was produced "for free" — verified in a pass that would otherwise make just one token. The one yellow correction per round is the target having the final say, which is what keeps output identical.
Act 3 · The dial that matters
Acceptance rate is everything
Speedup rides entirely on how often the draft agrees with the target. A draft that mimics the target well gets long accepted runs; a poor one gets rejected constantly. Tune the acceptance rate and the draft length, and watch the speedup respond.
Draft acceptance rate70%
Tokens drafted per round (K)4
—
Avg accepted / round
—
Speedup vs plain
—
Optimal K here
There's a sweet spot in K: draft too few and you underuse the free parallel pass; draft too many and you waste the small model on tokens that get rejected anyway. The best K rises as acceptance rises.
Why guessing is faster than working
The counterintuitive core: the big model still runs, yet you finish sooner. The resolution is that LLM decode is memory-bound. Producing one token means streaming every weight of the model through the GPU once; the actual arithmetic barely registers next to that memory traffic. So a forward pass has enormous spare compute — you're paying to move the weights whether you use that compute or not.
Speculative decoding spends that spare compute. Instead of asking the target "what's the next token?", you ask "here are 4 proposed tokens — for each position, what would you have said?" — and it answers all 4 in one weight-streaming pass. Every proposal it agrees with is a token you got without paying a second pass.
The lossless guarantee
It's not a quality gamble. The acceptance test uses the target's own probabilities, and rejected positions are re-sampled from the target — so the output is provably distributed exactly as if the target generated every token. You are not trading accuracy for speed. You're trading spare compute for it.
Where the draft comes from
The draft model just needs to be fast and usually right: a small model from the same family, a distilled version, or even the model's own extra prediction head (as in Medusa or the multi-token-prediction head DeepSeek-V3 trains). The closer the draft mimics the target, the higher the acceptance rate, the bigger the win.
The tuning
Two knobs. Acceptance rate is set by draft quality — better draft, longer accepted runs. K, the number of tokens drafted per round, has a sweet spot: too small underuses the free parallel verification, too large wastes draft effort on tokens likely to be rejected. Real systems see roughly 2-3× faster generation — pure latency, identical output.
Speculative decoding, step by step
The accept-or-resample rule is what makes the speedup free of any quality cost. Here it is as steps, then checked by hand on a three-token vocabulary, with the arithmetic behind the speedup numbers in Act 3.
One round of the algorithm
Draft. The small model q proposes K tokens one at a time, recording the probability q(x) it gave each.
Score in one pass. The target p runs once over the prefix plus all K drafts and returns its distribution at all K + 1 positions.
Accept left to right. Keep draft token x with probability min(1, p(x) / q(x)), and stop at the first rejection.
Repair or extend. At a rejection, sample a replacement from norm(max(0, p − q)), the target probability the draft under-covered. If all K pass, sample one bonus token from the target's last distribution. Each round therefore yields between 1 and K + 1 tokens.
Worked example: proof that nothing changes
Setup. Over tokens A, B and C the draft gives q = (0.6, 0.3, 0.1) and the target p = (0.4, 0.4, 0.2).
Acceptance. B and C are always kept, since p ≥ q for both. A is kept with probability 0.4/0.6 = 2/3, so a rejection happens 0.6 × 1/3 = 0.2 of the time.
Repair. max(0, p − q) = (0, 0.1, 0.1), normalized to (0, 0.5, 0.5).
Result. A: 0.6 × 2/3 = 0.4. B: 0.3 + 0.2 × 0.5 = 0.4. C: 0.1 + 0.2 × 0.5 = 0.2. Exactly the target's p, and the acceptance rate is α = Σ min(p, q) = 0.8.
Tokens per target pass are (1 − α^(K+1)) / (1 − α) when each draft is accepted with probability α: 3.36 for α = 0.8 and K = 4, 2.31 at α = 0.6, and 4.33 at α = 0.8 with K = 8.
Wall-clock speedup divides that by 1 + K·c, where c is one draft step's cost relative to a target pass. At c = 0.05 and α = 0.8, K = 4 gives 2.80× and K = 8, the best K, gives 3.09×. Act 3's defaults, α = 0.7 and K = 4, give 2.77 tokens per pass; with its draft cost of 0.12 that is 1.87× in wall-clock time, and K = 4 is the optimum.
Knobs and pitfalls
The draft must share the target's tokenizer, and the key-value cache entries for rejected tokens have to be rolled back. The gain is largest at small batch sizes, where decode is memory-bound; in a large batch the target is closer to compute-bound, verification stops being free, and the speedup shrinks. Measure α on your real traffic, since predictable text such as code or templated output is accepted more often than open-ended chat.
Draft sources trade quality for cost: a small model from the same family, extra prediction heads (Medusa, or DeepSeek-V3's multi-token prediction), a feature-level draft head (EAGLE), or no model at all, with n-gram prompt lookup that copies spans from the context, which works well for edits and retrieval answers. vLLM, SGLang and TensorRT-LLM all ship speculative decoding.
Check yourself
Four questions. No guessing — well, a little.
See every answer, with the reasoning
Why can the big model produce several tokens in one forward pass?
Answer: Decode is memory-bound, so a pass has spare compute to verify several drafted tokens at once — Streaming the weights once is the cost; verifying K guesses in that pass is nearly free compute.
Does speculative decoding change the output distribution?
Answer: No — rejected tokens are re-sampled from the target, so it is provably lossless — The target has the final say at every position, so output matches plain decoding exactly.
What most determines the speedup?
Answer: The acceptance rate — how often the draft agrees with the target — Better draft → longer accepted runs → bigger win. It is the dial that matters most.
What is a good source of a draft model?
Answer: A small same-family model, a distilled version, or the target’s own extra prediction head — The draft must be fast and usually right; MTP heads (e.g. DeepSeek-V3) do this in-model.