Autoregressive vs diffusion LLMs: one token at a time, or all at once?

An autoregressive (AR) model factorizes text as a chain: the probability of a sequence is the product of each token given all the tokens before it. So generation is inherently sequential — token t cannot be computed until token t−1 exists. A diffusion LLM instead starts from a fully masked or noised sequence and runs a fixed number of denoising steps, and at every step it updates all positions together using bidirectional attention. It trades "N tokens means N steps" for "K refinement passes over the whole thing, in parallel."

AR decode is famously memory-bandwidth bound: each token is a full forward pass that mostly moves weights and the KV cache through memory, and you cannot start the next one early. So wall-clock scales with token count. Diffusion decouples the two — if a model produces a good answer in, say, a dozen denoising steps regardless of how many tokens it emits, the sequential depth stops growing with length, and the parallel work per step maps well onto a GPU.

That is why reported throughput for diffusion LLMs has been eye-catching: vendor figures have put decode rates in the thousands of tokens per second (reported numbers around 1,000, 1,500, and 2,000+ tok/s on their hardware), against the tens-to-low-hundreds that a typical AR model streams. Treat those as reported, hardware- and setup-specific figures rather than a like-for-like law — but the direction is real, and it comes straight from replacing N sequential steps with K.

The historical catch is quality. AR training — predict the next token — matches how the model is used, and years of scaling have made it extremely good at long-range coherence. Early diffusion text models traded some of that away: refining all positions at once is a harder learning problem, and outputs could be less crisp on long, tightly-dependent passages. The gap has narrowed quickly, but for the highest-stakes open-ended generation, AR still tends to lead.

The pragmatic answer is not either/or. Block diffusion generates blocks of tokens autoregressively while denoising the tokens within each block in parallel — keeping AR-style long-range structure and KV-cache friendliness across blocks, while recovering diffusion’s parallel speed inside them. It is a direct attempt to take the latency win without paying the full coherence cost.

Imagine an inline coding assistant that must return a short completion the instant a developer pauses. The outputs are bounded (a line, a small block) and latency is everything — a perfect fit for a diffusion or block-diffusion decoder, where a fixed handful of steps returns the whole span at once instead of streaming it token by token. Shaving hundreds of milliseconds off time-to-answer here is worth a small quality concession.

Now the same product has a long-form "explain this codebase" mode: multi-paragraph, tightly reasoned, open-ended. That leans back toward autoregressive generation, where coherence over long spans and the mature streaming stack matter more than raw tok/s. Same product, two decoders, chosen by the shape of the output — which is exactly how this comparison tends to resolve in practice: not a winner, but a fit.