What is speculative decoding?

Speculative decoding speeds up LLM generation by using a small, fast draft model to guess several next tokens, then having the large target model verify all of them in a single parallel forward pass. Every token the target agrees with is kept; the first disagreement is corrected and the rest discarded. Because verification is parallel and decode is memory-bound, this yields 2-3x faster generation.

Does speculative decoding change the output?

No — it is lossless. The verification step uses the target model's own probabilities to accept or reject each drafted token, and a corrected token is sampled from the target whenever a draft is rejected. The result is provably distributed exactly as if the target model had generated every token itself. You get identical quality, only faster.

Why is it faster if the big model still runs?

Decode is memory-bound: generating one token requires streaming all the model's weights through the GPU, and the compute is nearly free by comparison. Verifying K drafted tokens in one pass streams the weights once instead of K times, so if most drafts are accepted you produce several tokens for the memory cost of one. The draft model's extra work is cheap because it is small.

What determines the speedup?

The acceptance rate — how often the draft model agrees with the target. A draft that closely mimics the target gets long accepted runs and large speedups; a poor draft gets frequent rejections and little gain. There is also a sweet spot in how many tokens to draft: too few wastes the parallel pass, too many wastes work on tokens likely to be rejected.

LABS · 16 / INFERENCE

The Understudy.

A giant model reads out one token at a time — slow, because each token drags all its weights through the GPU. So hire an understudy: a small fast model guesses the next few tokens, and the star checks them all in one pass, keeping every guess it agrees with. Same performance, a fraction of the passes.

Act 1 · Why decode is slow

One token, one whole model

Generating text token-by-token means one forward pass per token — and each pass streams every weight of a 70B model through the GPU. The compute is trivial; the memory movement is the wall. Watch the passes stack up.

Output (1 forward pass each)

Tokens produced

Forward passes

1.0

Tokens / pass

Standard autoregressive decode: tokens-per-pass is stuck at exactly 1. Every token pays the full memory cost of the model. Speculative decoding attacks precisely this ratio.

Act 2 · The trick

Draft, then verify in one pass

The draft model (small, fast) guesses the next 4 tokens. The target model checks all 4 in a single pass, accepting each while it agrees — until the first mismatch, which it corrects. Then the draft guesses again from there.

Draft model proposes

Target verifies → committed output

Tokens committed

Target passes

—

Tokens / pass

Every green token was produced "for free" — verified in a pass that would otherwise make just one token. The one yellow correction per round is the target having the final say, which is what keeps output identical.

Act 3 · The dial that matters

Acceptance rate is everything

Speedup rides entirely on how often the draft agrees with the target. A draft that mimics the target well gets long accepted runs; a poor one gets rejected constantly. Tune the acceptance rate and the draft length, and watch the speedup respond.

Draft acceptance rate70%

Tokens drafted per round (K)4

—

Avg accepted / round

—

Speedup vs plain

—

Optimal K here

There's a sweet spot in K: draft too few and you underuse the free parallel pass; draft too many and you waste the small model on tokens that get rejected anyway. The best K rises as acceptance rises.

Why guessing is faster than working

The counterintuitive core: the big model still runs, yet you finish sooner. The resolution is that LLM decode is memory-bound. Producing one token means streaming every weight of the model through the GPU once; the actual arithmetic barely registers next to that memory traffic. So a forward pass has enormous spare compute — you're paying to move the weights whether you use that compute or not.

Speculative decoding spends that spare compute. Instead of asking the target "what's the next token?", you ask "here are 4 proposed tokens — for each position, what would you have said?" — and it answers all 4 in one weight-streaming pass. Every proposal it agrees with is a token you got without paying a second pass.

The lossless guarantee

It's not a quality gamble. The acceptance test uses the target's own probabilities, and rejected positions are re-sampled from the target — so the output is provably distributed exactly as if the target generated every token. You are not trading accuracy for speed. You're trading spare compute for it.

Where the draft comes from

The draft model just needs to be fast and usually right: a small model from the same family, a distilled version, or even the model's own extra prediction head (as in Medusa or the multi-token-prediction head DeepSeek-V3 trains). The closer the draft mimics the target, the higher the acceptance rate, the bigger the win.

The tuning

Two knobs. Acceptance rate is set by draft quality — better draft, longer accepted runs. K, the number of tokens drafted per round, has a sweet spot: too small underuses the free parallel verification, too large wastes draft effort on tokens likely to be rejected. Real systems see roughly 2-3× faster generation — pure latency, identical output.

Check yourself

Four questions. No guessing — well, a little.

Finished this one? 0 / 61 Labs done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

LLM Engineering

The Understudy.

Why guessing is faster than working

Where the draft comes from

The tuning

Check yourself

Explore the topic

More Labs

Explore more from Vibe Engines

Why guessing is faster than working

Where the draft comes from

The tuning

Check yourself

Keep going

KV-Cache Visualizer →

GPU Fundamentals →

LLM Serving →

DeepSeek-V3 (MTP head) →

Explore the topic

More Labs

Explore more from Vibe Engines