Speculative decoding,
explained.

Rungs 2 and 3 make memory work harder. This rung attacks a waste that is visible only once you look at what generating one token costs. To emit a single token, the accelerator must read every weight in the model from memory and do very little arithmetic with each one. Decode is therefore memory-bandwidth-bound, and the chip's arithmetic units sit largely idle. Meanwhile, scoring several tokens at once costs almost exactly the same as scoring one, because the weight read is what you were paying for.

That asymmetry is an opportunity: the expensive model can check several candidate tokens in a single pass for nearly the price of producing one. So have a small, cheap model draft the next few tokens, then verify them all in one pass of the large model. Every draft token that survives is a token generated without its own weight read.

What makes this remarkable rather than merely clever is the acceptance rule. Tokens are accepted or rejected — and on rejection a replacement is sampled from a carefully-constructed residual distribution — such that the sequence emitted is distributed exactly as if the large model had generated it alone. Not approximately: provably identically. So the speed-up, reported in the region of two to three times, costs nothing in quality, which is close to unique among the optimisations on this site. How much you gain depends on how often the draft is right, and drafting further ahead helps until the compounding chance of an early rejection wastes the rest — so there is an optimal draft length.

A language model generates autoregressively: each token depends on all the ones before it, so they come out strictly one at a time. And here is the painful part — producing that single token requires reading the model's entire set of weights from memory. For a large model that is tens or hundreds of gigabytes moved per token. Decoding is memory-bandwidth-bound: the GPU's arithmetic units sit mostly idle, starved while weights stream in.

That idle compute is the opportunity. Checking whether a proposed token is correct costs one forward pass — and a forward pass can check many token positions at once for barely more than the cost of one, because it is bandwidth-bound, not compute-bound. So if something could guess the next few tokens, the big model could verify the whole batch of guesses in a single pass, spending its wasted compute to produce several tokens at once.

The scheme uses two models: a small fast draft model and the large target model whose output you actually want. Each round:

If the draft is good, most of its k guesses survive and you advance many tokens per expensive target pass. If it is bad, you reject early — but you have still produced at least one correct token (the target's own), so per target call you never do worse than ordinary decoding. (In wall-clock time a poor draft can still be a net loss, since you pay for the draft passes too.)

The subtle, beautiful part is that this is lossless. You might expect a shortcut to change what the model says — but speculative decoding is arranged so the tokens it emits are distributed exactly as if you had sampled from the target model normally. It uses a modified rejection sampling rule: accept the draft's token with probability min(1, p_target/p_draft), and on rejection sample the correction from an adjusted distribution. The algebra works out so the combined process's distribution equals the target's, token for token.

That guarantee is what makes it deployable. It is not an approximation with a quality knob — it produces the same samples as the model it accelerates, so a serving system can switch it on transparently and users see identical outputs, just faster. Speed for free, in the truest sense the field offers.

How much faster? Model each drafted token as accepted independently with probability α, the acceptance rate (how often the little model agrees with the big one). Accept a leading run until the first rejection, then add the target's one guaranteed token. The expected number of tokens produced per target pass is a geometric sum:

Standard decoding produces exactly 1 token per target pass, so this ratio is the speedup in target passes. With α = 0.8 and k = 4 it is ~3.4 tokens per pass. But drafting isn't free: each of the k draft tokens costs the draft model's (smaller) compute, wasted whenever the token is rejected. Dividing the expected tokens by that draft overhead gives a real-time speedup that peaks at a modest k and then falls — guessing too far ahead spends draft compute on tokens that rarely survive. The runnable model below computes both the token count and that peak.

Leviathan, Kalman and Matias use two models over the same vocabulary: the target M_p, whose output distribution p you want to sample from, and a cheaper draft M_q with distribution q. Their draft length is γ (the page's k). One call of the procedure, which the paper names SpeculativeDecodingStep, emits between 1 and γ + 1 tokens:

1. Draft γ tokens. Run the draft autoregressively from the current prefix: for i = 1…γ, compute q_i(x) = M_q(prefix + x₁…x_i−1) and sample x_i from it. Keep each full distribution q_i, not just the sampled token.
2. Score every position in one target call. Run M_p once, in parallel, on the prefix extended by all γ guesses. This yields γ + 1 distributions: p₁ for the first guess's position through p_γ+1 for the position after the last guess.
3. Accept or reject left to right. For each guess draw r_i uniformly from [0, 1] and accept x_i if r_i ≤ p_i(x_i)/q_i(x_i). A guess the target likes at least as much as the draft (p ≥ q) is always kept. Stop at the first rejection; call the number accepted n.
4. Correct, or take a bonus. If guess n + 1 was rejected, sample one replacement from the residual distribution norm(max(0, p_n+1 − q_n+1)). If all γ guesses survived, sample one extra token from p_γ+1, which the target call computed anyway.
5. Emit and repeat. Append x₁…x_n plus that one token to the prefix. Every call therefore yields at least one token drawn from the target's own distribution.

Why the result is exactly the target's distribution

Take one position, and let β = Σ_x min(p(x), q(x)) be the probability that the draft's token is accepted. The chance of emitting x through acceptance is q(x) · min(1, p(x)/q(x)) = min(p(x), q(x)). Rejection happens with probability 1 − β = Σ_x max(0, p(x) − q(x)), which is exactly the residual's total mass, and the residual then picks x with probability (p(x) − min(p(x), q(x)))/(1 − β). Adding the two routes gives min(p, q) + p − min(p, q) = p(x). Each later position repeats the argument conditioned on the accepted prefix, so the whole emitted sequence has the target's distribution.

The rule covers the usual decoding settings because temperature, top-k and nucleus sampling only reshape p and q, and the same reshaping can be applied to both before the test. Greedy decoding is the special case where both distributions are one-hot: a guess is kept exactly when it equals the target's argmax, and the correction is that argmax.

What it needs, and what came alongside

Nothing is retrained and the target's architecture is untouched; the draft only has to share the vocabulary. The paper tried drafts ranging from trivial unigram and bigram models to smaller Transformers. Chen and colleagues at DeepMind published the same accept-and-resample rule independently soon after as speculative sampling, speeding up Chinchilla 70B with a 4B draft by 2–2.5× in a distributed setup. Blockwise parallel decoding by Stern, Shazeer and Uszkoreit (2018) had verified several guessed tokens in one pass earlier, but only for greedy decoding.

• p, q — target and draft distributions at one position; DTV = ½ Σ|p − q|, the total variation distance between them.
• β, α — the acceptance probability at one position and its average over positions, which the analysis treats as independent.
• γ — draft length; c — the time of one draft step divided by the time of one target call.
• p′ — the residual distribution the correction is drawn from.

At the extremes: α → 1 gives γ + 1 tokens per call and an improvement of (γ + 1)/(γc + 1); α = 0 gives one token per call and an improvement of 1/(γc + 1), which is a slowdown. With a free draft (c → 0) the ceiling as γ grows is 1/(1 − α). The paper also shows that whenever α > c some γ helps, by at least (1 + α)/(1 + c) at γ = 1, and that total arithmetic rises: the method spends extra FLOPs, including on rejected guesses, to cut latency.

The main experiments sped up T5-XXL (11B parameters) in the T5X codebase, fine-tuned for English-to-German translation (WMT EnDe) and for CNN/DailyMail summarization, with T5-small (77M), T5-base (250M) and T5-large (800M) as drafts. Decoding ran on a single TPU-v4 at batch size 1, both greedily and by sampling at temperature 1, and the outputs matched standard decoding: exactly under greedy decoding, in distribution under sampling.

The draft-size ablation is the instructive part. Larger drafts agreed with T5-XXL more often, so α rose from T5-small to T5-large, but each draft step also cost more, and T5-large, the most accurate draft, gave the smallest speedups. The smallest draft won because its cost ratio c fell faster than its acceptance rate. Sampling at temperature 1 lowered α compared with greedy decoding, which is why the sampled rows are slower.

The paper also estimated α for pairs it did not time, including a small GPT-like decoder and LaMDA 137B with smaller LaMDA drafts, and found that even unigram and bigram drafts reach nonzero acceptance. Such a draft costs essentially nothing, so any acceptance rate α is worth up to 1/(1 − α), about 1.25× at α = 0.2. What the paper did not test is serving at large batch sizes, where the idle arithmetic the method spends is no longer idle.

The method needs a draft model that is well aligned with the target — a draft that rarely agrees gets rejected constantly and you have paid for it for nothing — so in practice it requires a small model from the same family, which is not always available and is extra memory when it is.

The limitation that matters most in production is one the framing makes easy to predict and easy to forget: the whole argument rests on decode being bandwidth-bound, and at large batch sizes it is not. Batching many sequences already amortises the weight read across all of them, so the arithmetic units are no longer idle and the free capacity speculation exploits has been spent. This helps most for low-concurrency, latency-sensitive serving — a single user waiting on a response — and least for a saturated throughput-oriented server.

How it held up: it is standard in every serious serving stack, and the exactness guarantee is why it was adopted without argument — there is no quality regression to weigh. The draft model's awkwardness is the one loose end, and rung 5 removes it.

Speculative decoding kicked off a wave of "guess-and-verify" inference methods, most aimed at removing the need for a separate draft model. Medusa bolts extra prediction heads onto the target model itself so it can propose its own future tokens. EAGLE drafts in the model's feature space for higher acceptance. Lookahead decoding generates and verifies n-grams without any draft model at all. All share the same skeleton: propose cheaply, verify in parallel, keep what's correct.

The lasting idea is a reframing of what a forward pass is for. A big model's pass has spare compute; instead of spending it on one token, spend it checking many. That principle — verification is parallel and cheap, generation is serial and expensive, so trade the former for the latter — now underlies a whole branch of efficient-inference research.

Speculative decoding is the key idea behind making LLM generation faster without changing what it produces. Normal decoding is one token per forward pass, and each pass is memory-bound, so the big model's compute sits mostly idle. Speculative decoding uses a small, cheap draft model to guess several tokens ahead, then has the large target model verify all those guesses in a single pass.

The guarantee is what makes it beautiful: the target model accepts the longest prefix of the draft that matches its own distribution and rejects the rest, and the acceptance/rejection sampling is designed so the final output is statistically identical to what the target model would have produced alone. You get a speedup for free — no quality loss, no different outputs.

How much you gain depends on how often the draft guesses right: a well-matched, fast draft model that agrees with the target most of the time yields several tokens per verification pass. The technique turns an underused, memory-bound big model into one that commits multiple tokens per step by spending a little cheap draft compute.

It matters because inference cost and latency dominate the economics of running LLMs, and lossless acceleration is rare and valuable. Speculative decoding — and its self-drafting variants like Medusa, plus draft-model and n-gram flavors — is now a standard feature of production serving stacks. It reframed the latency problem: don't shrink the model, predict its output cheaply and verify.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Five papers about the bill. Each one buys throughput back from a different part of the stack: the weights, the memory hierarchy, the KV cache, the decode loop.

1. 1Knowledge Distillation2014
2. 2FlashAttention2022
3. 3PagedAttention2023
4. 4Speculative Decoding2023
5. 5Medusa2024

All 110 papers, in 19 ladders →