LABS · 50  /  INFERENCE

All at once.

Every chatbot writes the same way: one token, then the next — the length of the answer is the latency. A text diffusion model refuses that. It starts from a block of pure mask and refines the whole thing at once over a handful of steps, deciding dozens of tokens per pass. Watch a block go from ██████ to clean, then tune the dial between speed and coherence.

Act 1 · The baseline
Autoregressive: one token per pass
An autoregressive model writes token t+1 only after token t exists — so N tokens = N forward passes, and latency grows with length. Watch the passes stack up, one per token.
Output (1 forward pass each)
0
Tokens produced
0
Forward passes
1.0
Tokens / pass
Standard autoregressive decode: tokens-per-pass is stuck at exactly 1. The chain is the cost. Diffusion attacks precisely this ratio — by deciding many positions in the same pass.
Act 2 · The trick
Start masked, refine in parallel
A diffusion decoder takes a block of 12 positions, starts them all masked (██), and runs K denoising steps. Each step looks at the whole partial block and commits more positions at once — until, after K passes, the block is clean. Step through it and watch many tokens land per pass.
The block · step 0 of 4
0
Tokens committed
0
Denoise passes
Tokens / pass
A 12-token block in 4 steps means 3 tokens per pass — 3× the autoregressive ratio, from the same model. The whole block is written in the passes an AR model would spend on 4 tokens.
Act 3 · The dial that matters
Fewer steps is faster — and riskier
Tokens per pass is B / K: bigger block or fewer steps means more speed. But tokens decided in the same step can't fully condition on each other, so committing too many at once costs coherence. Tune the block size and step count, and watch speed and coherence pull against each other.
Block size B32
Denoising steps K8
Tokens / pass (B/K)
Speed vs AR
Coherence

Why a whole block at once

Autoregressive generation is latency-bound by design: token t+1 conditions on token t, so you cannot start it until t exists. N tokens cost N sequential passes. Batching and speculative decoding soften this, but the shape — one token per step — stays.

Diffusion changes the shape. It treats a block of positions as something to sculpt from noise: start them all masked, then over K refinement steps predict the whole block at once, committing more positions each pass until it's clean. Emit B/K tokens per pass instead of one. That's where the reported 1,000+ tokens-per-second figures come from.

Text diffusion ≠ image diffusion

This shares a name with the image diffusion that powers Stable Diffusion — but text is discrete. You can't add 0.3 of Gaussian noise to the token "cat." So text diffusion corrupts by masking tokens and learns to un-mask them: the "noise level" is which positions are hidden, and denoising is progressive un-masking. Same reverse-a-corruption idea, categorically different data.

The speed–coherence dial

The step count K is a dial. Fewer steps commits more tokens per pass — faster, but tokens chosen in the same step can't condition on each other, so they can come out mutually inconsistent (locally fine, globally muddled). More steps restores that conditioning, and the coherence with it, at the cost of speed. At K = B you're deciding one token per step — that's just autoregression again, no free lunch.

Where it wins

Diffusion LLMs shine where raw speed beats the last few quality points — high-throughput drafting, latency-critical UX, cheap bulk generation. Autoregressive still tends to lead on the hardest, most sequential reasoning, where every token really does depend on the one before it. It's a new point on the speed-quality frontier, not a strict upgrade. Go deeper in the Diffusion LLMs handbook.

Check yourself

Four questions. Decode them all at once if you like.

Finished this one? 0 / 51 Labs done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Labs