Diffusion
LLMs.

Autoregressive (AR) models — GPT, Claude, Llama — generate token t+1 conditioned on tokens 1..t. That conditioning is the whole trick, and also the whole bottleneck: you cannot start token t+1 until token t exists. Generating N tokens means N sequential forward passes, so latency grows linearly with output length.

Batching and speculative decoding soften this, but the fundamental shape — one token per step — stays. Diffusion changes the shape.

A diffusion LLM takes a block of B positions, starts them all masked, and runs K refinement (denoising) steps. Each step looks at the whole partially-filled block and commits more tokens, until after K steps the block is clean. The key number: it emits B/K tokens per pass instead of one.

With a 64-token block refined in 8 steps, that's 8 tokens per pass — an 8× drop in pass count versus AR. Push K down and throughput climbs; the runnable below counts the passes for any B and K.

This shares a name with the image diffusion that powers Stable Diffusion and friends — but the mechanism underneath is different in a way that matters. Image diffusion adds and removes continuous Gaussian noise from pixel or latent values; "a little noise" is a real-valued perturbation of a real-valued signal.

Text is discrete. You can't add 0.3 of Gaussian noise to the token "cat" — there's no halfway between "cat" and "cot" in token space. So text diffusion uses a discrete corruption process: instead of noising values, it masks tokens and learns to un-mask them. The "noise level" is which positions are hidden, and denoising is progressive un-masking.

Same big idea — learn to reverse a corruption process, then run it from pure noise to generate — but "corruption" means something categorically different for tokens than for pixels.

One denoising step is one forward pass over the whole sequence: the prompt plus every response position, masked or not. The model predicts a token for every masked position at once. The sampler then decides which of those predictions to commit and which to throw back under the mask for the next step. That decision is the remasking strategy, and it matters as much as the model does.

Here is low-confidence remasking on a six-position answer to "What is the capital of France?", committing two tokens per step (illustrative):

Notice that the length was fixed before step 1. In LLaDA, generation length is a hyperparameter. The sampler allocates that many masked positions up front, the model learns during fine-tuning to fill the unused tail with end-of-sequence tokens, and everything after the first one is discarded. Every step still pays for the whole allocation, so reserving 1,024 positions for a 40-token answer burns most of the compute.

The schedule changes accuracy, not just speed. In the LLaDA paper, the 8B Instruct model scores 69.4 on GSM8K under pure diffusion sampling with low-confidence remasking, and 78.6 when it is sampled block by block. The authors trace the gap to "extensive EOS token padding in the SFT data, causing early termination in low-confidence remasking." The model is most confident about the padding, so it commits the ending first and squeezes the reasoning out. Forcing left-to-right blocks stops that.

An autoregressive model is cheap per step for a reason that has nothing to do with pass count: its KV cache. Under a causal mask, token 37 never attends to anything after it, so its keys and values are final the moment it is generated. Each new step computes one new position and reads the rest from memory.

Masked diffusion gives that up to get parallelism. The LLaDA paper says so directly: the model "does not use a causal mask," and as a result "LLaDA is incompatible with KV caching." Every position attends to every other position, the prompt included. When a step commits even one token, the hidden states of every position above the first layer shift, and so do their keys and values. Each denoising step is therefore a full forward pass over the prompt and the entire allocated response.

Illustrative: a 1,000-token prompt and a 256-token answer. AR does one 1,000-token prefill and then 256 one-token steps, about 1,256 positions computed. Full-sequence diffusion at 4 tokens per step runs 64 steps of 1,256 positions each, about 80,000. That's 4× fewer passes for roughly 64× more compute.

That trade can still win at small batch sizes. Single-stream AR decoding is bound by memory bandwidth: each step streams the full weights to produce one token while the compute units sit mostly idle, and a diffusion step puts that idle compute to work. The bill arrives under load, when a batched server is compute-bound and the recomputation eats throughput you could have served (see LLM Serving).

This is the real motive for block diffusion. Arriola et al.'s BD3-LMs (an ICLR 2025 oral) make the model autoregressive across blocks and diffusion within one. A finished block never changes again, so its keys and values are cached like an AR prefix, and each step recomputes only the block being denoised. The same move removes the fixed-length limit: the model keeps emitting blocks until one contains end-of-sequence.

The other route is to approximate. Fast-dLLM (Wu et al., May 2025) adds a training-free, block-wise approximate KV cache to LLaDA and Dream, reusing keys and values that barely move between adjacent steps. It pairs the cache with confidence-threshold decoding and reports up to 27.6× higher throughput with minimal accuracy loss.

The number of steps K is a dial, and it trades throughput against coherence. Fewer steps means more tokens committed simultaneously per pass — faster, but riskier: tokens chosen in the same step can't fully condition on one another, so they can come out mutually inconsistent (the classic diffusion failure where a sentence is locally fine but globally muddled). More steps restores that conditioning and the coherence with it, at the cost of speed.

The honest picture: diffusion LLMs win big where raw speed beats the last few quality points — high-throughput drafting, latency-critical UX, cheap bulk generation — while autoregressive still tends to lead on the hardest, most sequential reasoning, where every token really does depend on the one before it. It's a new point on the speed-quality frontier, not a strict upgrade.

Four systems carry most of what is public about diffusion language models. Every figure below comes from the builder's own paper or page.

The Gemini Diffusion row sums up the field: near-parity on code, where answers are short and checkable, and a real gap on knowledge- and reasoning-heavy benchmarks. Diffusion's wins show up where any-order generation helps. Dream beats autoregressive models of its size on Countdown and Sudoku, where the end constrains the start. On LLaDA's reversal test (completing the line before a given line of a Chinese poem), the 8B Instruct model scores 45.6 to GPT-4o's 34.3, while trailing badly in the forward direction (51.8 vs 82.7).

Each builder measured speed on its own hardware, so compare a diffusion figure with the autoregressive baseline in the same report, not with another vendor's number.

Inception's Mercury Coder Mini (1,109 tokens/sec on an H100) and Google's Gemini Diffusion (1,479 tokens/sec) both clear a thousand tokens per second, which is what you'd predict from emitting a block every handful of passes. That makes them compelling for interactive coding assistants, autocomplete at scale, and any workload where a human is watching the text stream in.

For an engineer the takeaway is a new axis to reach for: when latency dominates and the task tolerates a small quality trade, a diffusion decoder can be a step-change, not a tweak. Pair it with the image-diffusion foundations for the shared math, and with speculative decoding for the autoregressive world's answer to the same latency problem.