Speculative decoding,
explained.

A language model generates autoregressively: each token depends on all the ones before it, so they come out strictly one at a time. And here is the painful part — producing that single token requires reading the model's entire set of weights from memory. For a large model that is tens or hundreds of gigabytes moved per token. Decoding is memory-bandwidth-bound: the GPU's arithmetic units sit mostly idle, starved while weights stream in.

That idle compute is the opportunity. Checking whether a proposed token is correct costs one forward pass — and a forward pass can check many token positions at once for barely more than the cost of one, because it is bandwidth-bound, not compute-bound. So if something could guess the next few tokens, the big model could verify the whole batch of guesses in a single pass, spending its wasted compute to produce several tokens at once.

The scheme uses two models: a small fast draft model and the large target model whose output you actually want. Each round:

If the draft is good, most of its k guesses survive and you advance many tokens per expensive target pass. If it is bad, you reject early — but you have still produced at least one correct token (the target's own), so you never do worse than ordinary decoding, only better.

The subtle, beautiful part is that this is lossless. You might expect a shortcut to change what the model says — but speculative decoding is arranged so the tokens it emits are distributed exactly as if you had sampled from the target model normally. It uses a modified rejection sampling rule: accept the draft's token with probability min(1, p_target/p_draft), and on rejection sample the correction from an adjusted distribution. The algebra works out so the combined process's distribution equals the target's, token for token.

That guarantee is what makes it deployable. It is not an approximation with a quality knob — it produces the same samples as the model it accelerates, so a serving system can switch it on transparently and users see identical outputs, just faster. Speed for free, in the truest sense the field offers.

How much faster? Model each drafted token as accepted independently with probability α, the acceptance rate (how often the little model agrees with the big one). Accept a leading run until the first rejection, then add the target's one guaranteed token. The expected number of tokens produced per target pass is a geometric sum:

Standard decoding produces exactly 1 token per target pass, so this ratio is the speedup in target passes. With α = 0.8 and k = 4 it is ~3.4 tokens per pass. But drafting isn't free: each of the k draft tokens costs the draft model's (smaller) compute, wasted whenever the token is rejected. Dividing the expected tokens by that draft overhead gives a real-time speedup that peaks at a modest k and then falls — guessing too far ahead spends draft compute on tokens that rarely survive. The runnable model below computes both the token count and that peak.

The whole method lives or dies on the acceptance rate — how often the draft agrees with the target. Two levers set it, and both are design decisions:

Because acceptance varies with context, good implementations adapt: draft more tokens when recent acceptance was high, fewer when it drops. The payoff is real — production systems routinely report 2–3× faster generation, more on predictable content, at exactly zero cost to output quality.

Speculative decoding kicked off a wave of "guess-and-verify" inference methods, most aimed at removing the need for a separate draft model. Medusa bolts extra prediction heads onto the target model itself so it can propose its own future tokens. EAGLE drafts in the model's feature space for higher acceptance. Lookahead decoding generates and verifies n-grams without any draft model at all. All share the same skeleton: propose cheaply, verify in parallel, keep what's correct.

The lasting idea is a reframing of what a forward pass is for. A big model's pass has spare compute; instead of spending it on one token, spend it checking many. That principle — verification is parallel and cheap, generation is serial and expensive, so trade the former for the latter — now underlies a whole branch of efficient-inference research.