Medusa,
explained.

Recall the bottleneck: language models generate one token at a time, and each token forces the whole model to be read from memory, leaving the hardware's compute mostly idle. Speculative decoding fills that idle compute by having a small, fast draft model guess several tokens ahead, which the big model then verifies in parallel. It works — but the draft model is a real operational cost. You have to find or train one that is both cheap and well-aligned with the target, serve it alongside, and maintain the pairing as models change.

Medusa removes the second model. Its insight: the big model already computes a rich hidden state at each position — enough, with a little extra machinery, to predict not just the next token but the one after, and the one after that. So instead of a separate drafter, give the model a few lightweight heads that read that hidden state and guess further ahead. The model drafts for itself.

A standard model has one output head predicting the next token. Medusa adds K more, each a small feed-forward layer on the same final hidden state, trained to predict the token at position +2, +3, …, +K+1. They are cheap to add (the backbone is frozen; only the heads train) and cheap to run (a few extra small layers). In one forward pass, the model now emits its usual next token plus K guesses for the tokens beyond it.

The catch is accuracy. Predicting the very next token is relatively easy; predicting five tokens ahead, with all the uncertainty of the words in between, is much harder. So head accuracy fades with distance — the first head is fairly reliable, later heads much less so. That decay is exactly what governs how many tokens Medusa can actually keep per pass, and it's the number the math below tracks.

Because the far heads are unsure, betting everything on their single top guess is wasteful — often it's wrong, and the whole speculation stops there. Medusa hedges: each head proposes its top few candidates, and the combinations form a tree of possible continuations (this head's option A followed by that head's options X or Y, and so on). A single guess per head accepts if it's right with probability a; offering s candidates accepts if any of them is right — probability 1 − (1 − a)^s, which climbs fast.

The clever part is verifying this whole tree in one forward pass. Medusa builds a special attention mask — tree attention — so that each candidate token only attends to its own ancestors in the tree, letting many candidate paths be scored simultaneously without interfering. One pass, many hypotheses checked, and the longest fully-correct path is accepted. It is how Medusa turns uncertain heads into reliable speedups.

Verification keeps the output honest. The tree of candidates is scored by the true model, and Medusa accepts the longest path whose tokens the model agrees with — falling back to the guaranteed correct next token when the heads miss. With greedy acceptance, a position is only kept if every head before it was also right, so each head's contribution is the cumulative product of accuracies. Summing those, plus the always-correct base token, gives the expected tokens per pass:

Because each term multiplies one more (sub-1) accuracy, later heads add less and less — a fading geometric-like series. With four heads at accuracies 0.8/0.65/0.5/0.35 the model produces ~2.7 tokens per pass instead of 1; perfect heads would give K+1. Tree candidates raise each head's effective accuracy toward 1, pushing the sum higher. Crucially, the accepted tokens are exactly the model's own, so — like draft-model speculation — the output is lossless. The runnable version below computes both formulas.

Medusa and draft-model speculative decoding are two routes to the same goal; each has a personality:

Where a well-matched draft model can sometimes propose longer accurate runs, Medusa wins on operational simplicity — one model, a handful of heads, no second system. For many deployments that trade is exactly right.

Medusa is part of a fast-moving family of self-speculative decoding methods — including EAGLE, Hydra, and lookahead decoding — all aiming to get speculative decoding's speed without the burden of a separate draft model. They differ in how they draft (extra heads, feature-space prediction, n-gram pools) but share the skeleton established by speculative decoding: propose cheaply, verify in parallel, keep what's correct.

Its contribution to that lineage is a clean, practical recipe: reuse the model's own hidden state through lightweight heads, and hedge their uncertainty with tree attention. That combination — self-drafting plus batched candidate verification — turned out to be a sweet spot of simplicity and speed, and it's why Medusa became a widely-used, easy-to-adopt inference accelerator.