Classifier-free
guidance.

A diffusion model conditioned on text will follow the prompt — loosely. Samples drift toward the general shape of the data rather than the specific thing asked for, because nothing in the objective rewards obedience over plausibility. The existing fix was classifier guidance: train a separate classifier that works on noisy images, and use its gradient to push each denoising step toward the desired class. It works, and it requires training and maintaining a noise-aware classifier, which is a second model and an awkward one.

The change removes it, and the trick is to get two models out of one. During training, drop the condition some fraction of the time — replace the prompt with nothing — so the same network learns to denoise both conditionally and unconditionally. At sampling time, run it twice and compare: the difference between the conditional prediction and the unconditional one is the direction that the prompt is asking for. Then extrapolate along it, going further than the conditional prediction alone, by a factor you choose.

That factor is the guidance scale, and stated in units, it is the single knob every user of every image model turns. At one, you get ordinary conditional sampling. Higher, and samples obey the prompt more insistently. The cost is exactly one thing and worth knowing: each step now costs two network evaluations instead of one, because the model must be run with and without the condition.

A diffusion model generates by starting from pure noise and, step by step, predicting and removing a little noise until an image emerges. To make it follow a prompt, you condition that noise prediction on the text. But conditioning alone is often too gentle — the model drifts, hedges, produces something loosely on-topic. You want a dial: how hard should it obey the prompt?

The first answer, classifier guidance, borrowed a separate image classifier and pushed each denoising step toward whatever the classifier thought matched the label — effective, but it meant training and running an extra network on noisy images, an ungainly failure-prone add-on. Classifier-free guidance (Ho & Salimans, 2021) got the same dial with no extra model at all. The trick: train the one diffusion model to work both with the prompt and without it, then at sampling time ask it for both predictions and combine them. That combination is the guidance knob, and it's now in essentially every text-to-image and text-to-video system.

The prerequisite is a diffusion model that can run both ways. You get it almost for free during training: with some probability, randomly drop the condition — replace the text prompt with an empty/null token — so the same network learns to denoise both when told a prompt and when told nothing. After training, one network gives you two predictions on demand: ε_cond (the noise it expects given your prompt) and ε_uncond (the noise it expects given no prompt at all).

These two predictions point in slightly different directions. The unconditional one heads toward "any plausible image." The conditional one heads toward "a plausible image that matches the prompt." The difference between them, ε_cond − ε_uncond, is therefore the conditional direction — the part of the prediction that is specifically because of the prompt. Guidance is nothing more than deciding how far to travel along that difference vector. The whole method reduces to: get both predictions, take their difference, and amplify it.

Guidance starts from the unconditional prediction and steps past the conditional one — an extrapolation, not an interpolation. That "overshoot" is what makes prompt adherence stronger than plain conditioning:

Read it as: begin at the unconditional prediction, then travel (1+w) times the conditional-direction vector. At w = 0 you land exactly on ε_cond — normal conditioning. As w grows you shoot beyond it, exaggerating the very features the prompt is responsible for, which is why higher guidance gives tighter, more literal prompt-following. But it's an extrapolation, and extrapolating too far leaves the region where the model's predictions are reliable — hence the familiar failure of high guidance scales: blown-out colors, harsh contrast, and artifacts. The whole art of setting the slider is finding how far you can push before the extrapolation breaks.

The guided noise prediction is a weighted combination of the two, with guidance weight w:

w = 0 gives ε_cond (plain conditioning); w = −1 gives ε_uncond (no prompt). For ε_cond = 2, ε_uncond = 1: at w = 3, ε = 4·2 − 3·1 = 5 — pushed well past the conditional value of 2.

The same thing, written as an extrapolation, makes the geometry explicit — start at the unconditional prediction, travel (1+w) along the conditional direction:

Identical to the first form — expand it and the ε_uncond terms recombine. The bracket ε_cond − ε_uncond is the conditional direction; (1+w) is how far you go. The runnable version below computes the guided prediction, shows the two forms agree, and sweeps the weight.

Ho and Salimans start from classifier guidance (Dhariwal and Nichol, 2021), which mixes a diffusion model's score with the gradient of a separately trained classifier on noisy images, and ask whether the same effect, trading diversity for fidelity the way truncation does in GANs, can be had with no classifier at all. The answer has a training half and a sampling half. The paper works in continuous time, with noisy inputs z_λ = α_λ x + σ_λ ε indexed by the log signal-to-noise ratio λ, but the recipe carries over unchanged to discrete-step models.

1. Give the model a null condition. Use one network ε_θ(z_λ, c) and reserve a token ∅ meaning "no condition". The unconditional model is ε_θ(z_λ, ∅); there is no second set of weights.
2. Train with condition dropout. For each training example, with probability p_uncond replace the class label c by ∅, then take an ordinary diffusion step: noise the image to a random λ and regress the noise. The paper swept p_uncond over 0.1, 0.2 and 0.5.
3. Run the network twice per step. At every sampling step, evaluate ε_θ(z_λ, c) and ε_θ(z_λ, ∅) on the same noisy input. Implementations usually batch the two, but it is still twice the compute.
4. Combine. Form ε̃ = (1 + w) ε_θ(z_λ, c) − w ε_θ(z_λ, ∅) and hand ε̃ to the usual sampler in place of the conditional prediction. The paper swept w from 0 up to 4.
5. Change nothing else. The experiments reused the architectures and hyperparameters of Dhariwal and Nichol's class-conditional ImageNet models at 64×64 and 128×128, even though they were tuned for classifier guidance, and altered only the training dropout and the sampling rule.

The convention trap: w versus guidance scale

The paper's weight w measures extra push beyond the conditional prediction, so w = 0 is plain conditional sampling. Most later papers and libraries, including GLIDE, Imagen and the diffusers implementation of Stable Diffusion, write the same rule as ε_∅ + s(ε_c − ε_∅) with a guidance scale s = 1 + w. Under that convention s = 1 is plain conditional sampling and s = 0 ignores the condition. A guidance scale of 7.5, the familiar Stable Diffusion default, is w = 6.5 in this paper's notation. Mixing the two conventions is the most common way to misread a guidance number.

Why it works without a classifier

Bayes' rule gives p(c | z) ∝ p(z | c)/p(z), so the gradient of an implicit classifier's log-probability is the conditional score minus the unconditional score, and a diffusion model's noise prediction is a scaled score. Classifier guidance with this implicit classifier therefore needs only the two predictions the model already makes. The authors add their own caveat: the networks are not exact scores, so the guided prediction is not, in general, the gradient of any actual classifier. The sampling target it approximates is p(z | c) times the implicit classifier's probability raised to the power w.

What the dropout rate controls

p_uncond sets the share of training spent on the unconditional task. A small share proved enough: 0.1 and 0.2 performed about the same, and 0.5 was worse.

• p(c | zλ) — the implicit classifier, never trained or evaluated; only its gradient appears, as a difference of two predictions.
• σλ — the noise scale at this level; dividing by it turns a noise prediction into a score.
• εθ(zλ) — the paper's shorthand for the unconditional prediction ε_θ(z_λ, ∅).
• w and s — the paper's weight (w = 0 is the conditional model) and the libraries' scale (s = 1 is the conditional model).

Extremes: w = −1 (s = 0) gives the unconditional model and w = 0 (s = 1) the conditional one. Written without the classifier, the target is p(z | c)^^1+w / p(z)^^w: the conditional model raised to a power above one and divided by the unconditional one, so as w grows it concentrates on the inputs that look most like the class, which is why diversity falls.

All experiments are class-conditional ImageNet at 64×64 and 128×128, using the architectures and hyperparameters of Dhariwal and Nichol's guided diffusion models, so the comparison with classifier guidance is like for like. Two knobs were swept, the training dropout p_uncond and the guidance weight w, with samples scored by FID (lower is better) and Inception Score (higher is better). The question was whether the FID–IS trade-off curve that classifier guidance and GAN truncation produce could be had without a classifier.

The weight sweep is the main result and also the main negative one. A little guidance improves FID over the unguided model, because samples become more convincing faster than they lose variety. Beyond a few tenths, FID gets worse while Inception Score keeps climbing: samples look more like their class and less like the full data distribution. No single w optimizes both, which is why guidance became a user-facing setting rather than a fixed constant. The p_uncond result is quieter but useful: the unconditional model needs only a small slice of training, so the method costs almost nothing in capacity.

The paper names its own limits. Sampling runs the network twice per step, which it suggests might be reduced by architectures that inject the condition late, so the two passes share most of their computation. And any method that buys fidelity with diversity raises the question of whether the lost diversity is acceptable for a given use. Later, Meng and colleagues (2023) distilled guided models into a single pass per step.

The trade is named in the paper and is visible in every image anyone has generated: guidance buys fidelity to the prompt at the cost of diversity. Turn it up and samples converge on a narrow, high-contrast, often over-saturated interpretation; turn it down and you get variety, including variety you did not want. There is no setting that is correct, which is why it is exposed to users.

The mechanism also explains why the artefacts look the way they do. The step is an extrapolation — it deliberately moves beyond what the model predicts — so at high scales it can leave the region the model was trained on entirely, and the results are not merely wrong but wrong in a characteristic, over-cooked way. Later work on clamping and rescaling the guided prediction is all about staying on the manifold while keeping the obedience.

How it held up: universally adopted, immediately, and still the default. It is the clearest case on this ladder of a technique whose contribution is control rather than capability — the model was already able to produce the image, and this is what lets someone ask for it. Worth noting how cheap the idea was: train with the condition dropped occasionally, which costs nothing, and the ability to dial conditioning strength falls out of arithmetic on two predictions.

Classifier-free guidance is now the default way conditional diffusion models are steered. The "guidance scale" (CFG scale) in Stable Diffusion and every mainstream text-to-image tool is a rescaling of the w in this formula (tools usually expose s = 1 + w), and the same idea is widely used in text-to-video diffusion models. A huge fraction of the practical controllability people enjoy in image generation — the ability to say "follow my prompt more" and have it work — is this one weighted subtraction.

Its ubiquity also made its main cost worth attacking. Because guidance doubles the forward passes per step, later work on guidance distillation trained models to reproduce the guided output in a single pass, folding the effect into the network so you keep the control without the 2× compute — an important optimization for fast and real-time generation. Variants also refine where guidance is applied (only at certain noise levels) or how it's scheduled across the sampling trajectory. But the core object — two predictions, extrapolate along the difference — is remarkably stable, and it's one of those rare ideas that is both a one-line formula and a cornerstone of a whole generation of creative tools.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Ten papers from the first latent-variable generator to video and real-time 3D. The diffusion half is strictly cumulative — each rung removes a cost the last one paid.

1. 1VAE2014
2. 2GANs2014
3. 3VQ-VAE2017
4. 4DDPM2020
5. 5DDIM2021
6. 6Classifier-Free Guidance2021
7. 7Latent Diffusion2022
8. 8Sora2024
9. 9NeRF2020
10. 103D Gaussian Splatting2023

All 110 papers, in 19 ladders →