Speculative decoding pairs a cheap draft model with the target model. The draft proposes a few tokens; the target checks them all in a single forward pass, keeping the longest correct prefix. Accepted tokens cost one big-model pass for many tokens; rejected ones fall back safely. Output distribution is provably unchanged — it is pure latency win, typically 2–3× on decode.
Worked example: a small fast ‘draft’ model proposes several tokens ahead, and the big model verifies them in one parallel forward pass — accepting the run up to the first disagreement — so you get multiple tokens per expensive step without changing the output distribution. Gotcha: the speedup depends on the draft’s acceptance rate: if the draft rarely matches, you pay for the draft AND the verify with little gain; it works best when the draft is well-aligned and the text is predictable, and it never changes the final result (verification guarantees exactness).