Every chatbot writes the same way: one token, then the next — the length of the answer is the latency. A text diffusion model refuses that. It starts from a block of pure mask and refines the whole thing at once over a handful of steps, deciding dozens of tokens per pass. Watch a block go from ██████ to clean, then tune the dial between speed and coherence.
An autoregressive model writes token t+1 only after token t exists — so N tokens = N forward passes, and latency grows with length. Watch the passes stack up, one per token.
Output (1 forward pass each)
0
Tokens produced
0
Forward passes
1.0
Tokens / pass
Standard autoregressive decode: tokens-per-pass is stuck at exactly 1. The chain is the cost. Diffusion attacks precisely this ratio — by deciding many positions in the same pass.
Act 2 · The trick
Start masked, refine in parallel
A diffusion decoder takes a block of 12 positions, starts them all masked (██), and runs K denoising steps. Each step looks at the whole partial block and commits more positions at once — until, after K passes, the block is clean. Step through it and watch many tokens land per pass.
The block · step 0 of 4
0
Tokens committed
0
Denoise passes
—
Tokens / pass
A 12-token block in 4 steps means 3 tokens per pass — 3× the autoregressive ratio, from the same model. The whole block is written in the passes an AR model would spend on 4 tokens.
Act 3 · The dial that matters
Fewer steps is faster — and riskier
Tokens per pass is B / K: bigger block or fewer steps means more speed. But tokens decided in the same step can't fully condition on each other, so committing too many at once costs coherence. Tune the block size and step count, and watch speed and coherence pull against each other.
Block size B32
Denoising steps K8
—
Tokens / pass (B/K)
—
Speed vs AR
—
Coherence
Why a whole block at once
Autoregressive generation is latency-bound by design: token t+1 conditions on token t, so you cannot start it until t exists. N tokens cost N sequential passes. Batching and speculative decoding soften this, but the shape — one token per step — stays.
Diffusion changes the shape. It treats a block of positions as something to sculpt from noise: start them all masked, then over K refinement steps predict the whole block at once, committing more positions each pass until it's clean. Emit B/K tokens per pass instead of one. That's where the reported 1,000+ tokens-per-second figures come from.
Text diffusion ≠ image diffusion
This shares a name with the image diffusion that powers Stable Diffusion — but text is discrete. You can't add 0.3 of Gaussian noise to the token "cat." So text diffusion corrupts by masking tokens and learns to un-mask them: the "noise level" is which positions are hidden, and denoising is progressive un-masking. Same reverse-a-corruption idea, categorically different data.
The speed–coherence dial
The step count K is a dial. Fewer steps commits more tokens per pass — faster, but tokens chosen in the same step can't condition on each other, so they can come out mutually inconsistent (locally fine, globally muddled). More steps restores that conditioning, and the coherence with it, at the cost of speed. At K = B you're deciding one token per step — that's just autoregression again, no free lunch.
Where it wins
Diffusion LLMs shine where raw speed beats the last few quality points — high-throughput drafting, latency-critical UX, cheap bulk generation. Autoregressive still tends to lead on the hardest, most sequential reasoning, where every token really does depend on the one before it. It's a new point on the speed-quality frontier, not a strict upgrade. Go deeper in the Diffusion LLMs handbook.
Masked diffusion decoding, step by step
How a masked-diffusion language model fills a block, a hand-run of an eight-token block in four steps, and the speed arithmetic behind the widget's two sliders. The confidences below are invented for illustration, not taken from a real model.
One block, start to finish
Start fully masked. Fill the block of B positions with a mask token after the prompt.
Predict everywhere. One forward pass gives every masked position a distribution over tokens, and a most-likely token with a confidence.
Commit the surest. Keep the B/K most confident predictions and leave the rest masked. Choosing by confidence, rather than left to right, is called low-confidence remasking.
Repeat. Each pass now conditions on the committed tokens too. After K passes nothing is masked.
Training is the reverse: mask each token with a random probability t, ask the model to restore the originals, and weight the loss on the masked positions by 1/t.
Worked example: B = 8, K = 4
Rate. B / K = 2 tokens per pass, so the block costs 4 passes where autoregression needs 8. That is a 2× reduction in passes, before counting that each diffusion pass reads the whole block.
Pass 1. All eight positions are masked. Suppose the confidences are 0.95, 0.91, 0.70, 0.62, 0.55, 0.50, 0.44 and 0.30 for positions 1 to 8. Commit positions 1 and 2.
Pass 2. With those two fixed, positions 3 to 8 are re-predicted; say position 3 rises to 0.93 and 4 to 0.88, and both are committed.
Passes 3 and 4. Positions 5 and 6, then 7 and 8, are committed the same way. Each pass fixes two tokens that had been uncertain, because the surrounding words are now known.
Where it fails. If positions 5 and 6 each had two near-equal candidates, committing both in one pass can pair incompatible words, such as two different subjects, since neither saw the other's choice. Dropping to K = 8, one token per pass, removes the conflict and the speedup together.
At the widget's defaults B = 32 and K = 8, that is 4 tokens per pass and a 4× reduction in passes; B = 32 with K = 32 is autoregression again.
Knobs and pitfalls
Tune K against a quality metric on your own task, not a fixed ratio. Long outputs are generated block by block (semi-autoregressive), which allows variable length and lets earlier blocks be cached; bidirectional attention inside a block makes ordinary key-value caching harder than in autoregressive models. Wall-clock gains depend on batch size and hardware, so measure end-to-end latency rather than counting passes.
Check yourself
Four questions. Decode them all at once if you like.
See every answer, with the reasoning
Why can a diffusion LLM emit several tokens per forward pass?
Answer: It refines a whole block in parallel over K steps, committing B/K tokens per pass instead of one — The block is decided together over a few denoise steps, so each pass lands multiple positions.
How does text diffusion differ from image diffusion?
Answer: Text is discrete, so it masks tokens and un-masks them instead of adding/removing Gaussian noise — You can't add a little Gaussian noise to a token — the "noise" is which positions are masked.
What happens at K = B (steps equal block size)?
Answer: It reduces to autoregression — one token per step, full coherence, no speedup — One position per step is exactly sequential decoding; the diffusion win needs K < B.
What is the trade-off of using very few steps?
Answer: More tokens per pass (faster) but lower coherence, since co-committed tokens can't condition on each other — Speed rises as B/K rises, but parallel-commitment error grows with it.