DDPM,
explained.

Rung 2's difficulty is that both halves of the game are learned, so nothing is stable. This rung's change is to make one direction fixed, simple and known in advance: define a process that takes an image and adds a little Gaussian noise, over and over, for many steps, until nothing is left but noise. That direction involves no learning at all — it is a schedule someone wrote down.

All the model has to learn is to undo one step of it. And because each forward step is a small Gaussian perturbation, the corresponding reverse step is also approximately Gaussian, which makes it something a network can be asked to predict. Generation is then: start from pure noise, apply the learned reverse step many times, arrive at an image.

The training objective is where this becomes remarkable. After simplification, what the network is asked to do is: given a noisy image and how noisy it is, predict the noise that was added — and the loss is a plain squared error on that prediction. Generative modelling has become regression. There is no adversary, so no saddle point and no instability; the loss is a number that decreases and means something; and because every training example is a real image with noise on it, there is no mechanism for mode collapse — the model never gets to choose which parts of the distribution to cover. Stated against rung 2, that is the whole argument, and it is why this family won.

Imagine dropping ink into water. It disperses until the glass is a uniform haze — an easy, predictable process to run forward. Now imagine running it backward: reassembling that haze into the original drop. Impossible to do exactly, but if you learned, at every tiny moment, how to nudge the haze one step back toward order, you could reverse the whole thing.

That's diffusion. The forward direction gradually destroys an image into noise — trivial. The reverse direction rebuilds an image from noise — hard, and exactly what we train a network to do. Generate by starting from pure noise and running the learned reverse steps until a picture appears.

The forward process is a fixed Markov chain with no learnable parameters. Over T steps (the paper uses T = 1000), it repeatedly adds a little Gaussian noise. A noise schedule β₁ … β_T — small numbers that grow over time — sets how much noise each step injects:

In words: each step shrinks the current image slightly and sprinkles in fresh noise. Start with a clean image x₀; after enough steps you reach x_T, which is indistinguishable from pure Gaussian noise. This process is a given — its only purpose is to define the corruption the network must learn to undo.

Adding noise a thousand times to make one training example would be painfully slow. A neat property of Gaussians rescues it: you can jump straight to any timestep in one shot. Define α_t = 1−β_t and ᾱ_t = α₁·α₂·…·α_t, and then

So to make a training example at noise level t, you take a clean image, pick a random t, sample one blob of noise ε, and combine them by the formula above — no thousand-step loop needed. This closed form is what makes DDPM training efficient, and it's the key to the objective coming up.

Reversing the chain means going from x_t to a slightly cleaner x_{t-1}. The true reverse step is intractable, but if each forward step adds only a little noise, each reverse step is well-approximated by a Gaussian too — so we train a network to predict its mean.

That network is a U-Net: an encoder-decoder with skip connections, well suited to images, that also takes the timestep t as input so it knows how noisy things currently are. Apply it repeatedly — x_T → x_{T-1} → … → x₀ — and noise becomes an image. All the model's intelligence lives in this one denoising network.

Earlier diffusion work derived a complicated variational bound and tried to predict full distributions — fiddly and unstable. DDPM's decisive move was to reparameterize the problem. Since x_t = √(ᾱ_t)x₀ + √(1−ᾱ_t)ε, knowing the noise ε is equivalent to knowing the clean image. So just predict ε. The training loss collapses to plain mean-squared error:

The recipe is almost embarrassingly simple: take a clean image, add a known amount of noise, and ask the network to guess the noise you added. Penalize the squared error. That's it. This "simple objective" — a reweighted version of the variational bound — produced dramatically better samples than the true bound (the fixed-variance bound itself trained without trouble; the gain was in sample quality) — the entire reason diffusion suddenly worked.

The paper's exact recipe: T = 1000 steps, β rising linearly from 0.0001 to 0.02. Everything interesting lives in the cumulative product ᾱ_t = Π(1−β_s) — the fraction of original signal surviving at step t. Compute it:

Two things to notice. The destruction is front-loaded gently — after 100 of 1000 steps, 95% of the signal amplitude survives; the schedule spends most of its budget in the middle. And at t = 1000 the signal fraction is 0.6% — which is why sampling can start from pure random noise: the endpoint of the forward process genuinely is one.

The reparameterization the whole paper turns on is one line of algebra on the shortcut formula. If you know the noise, you know the image:

Training teaches a network to estimate ε; the identity above converts that estimate back into an image, one denoising step at a time. Generative modeling became regression on noise.

Ho, Jain and Abbeel's Denoising Diffusion Probabilistic Models took the diffusion probabilistic model of Sohl-Dickstein and colleagues (2015), chose a specific parameterization of its reverse process, and showed that the training loop then shrinks to three lines. The two procedures below are the paper's training and sampling algorithms in full.

Training

1. Prepare data and schedule. Scale pixel values to [−1, 1]. Fix T = 1000 and a linear β schedule from 10^⁻⁴ to 0.02, and precompute α_t = 1 − β_t and ᾱ_t = α₁⋯α_t.
2. Draw one training example. Take an image x₀, a timestep t uniformly from 1 to T, and noise ε ~ N(0, I). Form x_t = √ᾱ_t x₀ + √(1 − ᾱ_t) ε in one shot, with no chain to run.
3. Regress the noise. Take a gradient step on ‖ε − ε_θ(x_t, t)‖^², and repeat until converged.

Sampling

1. Start from noise. Draw x_T ~ N(0, I).
2. Step down one level. For t = T down to 1, compute (1/√α_t)(x_t − (β_t/√(1 − ᾱ_t)) ε_θ(x_t, t)) and add σ_t z with fresh z ~ N(0, I), except at the last step, which adds no noise.
3. Fix the variance. The paper does not learn σ_t^². It tried β_t and the posterior variance β̃_t = β_t(1 − ᾱ_t−1)/(1 − ᾱ_t) and got similar results. The first is optimal if the data were standard normal noise, the second if the dataset were a single point, so the two bracket the right answer.
4. Output. After the last step, x₀ is the sample. Each of the 1,000 steps is one full network evaluation.

The network and the training run

ε_θ is a U-Net in the style of an unmasked PixelCNN++, with group normalization, two residual blocks per resolution, self-attention at the 16×16 feature-map resolution, and the timestep injected through a Transformer-style sinusoidal embedding; one set of weights serves every t. The CIFAR-10 model had 35.7 million parameters, the 256×256 LSUN and CelebA-HQ models 114 million, and a larger LSUN Bedroom variant about 256 million. Training used Adam at a learning rate of 2×10^⁻⁴ (2×10^⁻⁵ at 256×256, where the higher rate was unstable), batch size 128 for CIFAR-10 and 64 for the larger images, an exponential moving average of the weights with decay 0.9999, random horizontal flips on CIFAR-10, and dropout 0.1 on CIFAR-10, without which samples showed overfitting artifacts. The CIFAR-10 model trained for 800,000 steps on a TPU v3-8 in about 10.6 hours, and a batch of 256 samples took about 17 seconds.

Why the last step is special

The final term of the bound is the likelihood of the discrete pixel values given x₁. The paper models it with a discrete decoder: a Gaussian at the predicted mean, integrated over each pixel's bin of width 2/255, with the outermost bins extended to infinity. That keeps the reported likelihood a true bound in bits per dimension; at sampling time the predicted mean is shown without noise.

• LT — the first KL has no parameters, since β is fixed, so it is a constant.
• q(xt−1 | xt, x0) — the forward process run backward once x₀ is known: Gaussian, closed form, so every L_t−1 is a KL between two Gaussians.
• μθ — written in the same form as μ̃_t, with x₀ replaced by its estimate from ε_θ, so the distance between means becomes a scaled error on ε.
• Lsimple — drops the weight in front and samples t uniformly.

The dropped weight is far from constant. With σ_t^² = β_t it equals β_t/(2α_t(1 − ᾱ_t)): about 0.5 at t = 1 and about 0.01 at t = 1000. Setting it to 1 everywhere down-weights the easy, nearly clean steps relative to the bound and spends effort on heavily corrupted ones, which is why L_simple gives better samples and a slightly worse likelihood.

The headline benchmark was unconditional CIFAR-10, scored by Inception Score (IS, higher is better), FID (lower is better) and negative log-likelihood in bits per dimension; 256×256 models were trained on CelebA-HQ and on LSUN Bedroom, Church and Cat. The decisive experiment is an ablation of what the network predicts and which loss trains it, on CIFAR-10. Dashes mark runs the paper reports as unstable, with poor samples.

Read across the rows. With fixed variances and the true bound, predicting ε and predicting μ̃ give about the same quality (FID 13.51 against 13.22), and learning the variances either hurt or made training unstable. The gain comes from the reweighted L_simple, which needs the ε parameterization. The resulting IS 9.46 and FID 3.17 were, in the authors' words, better than most models in the literature at the time, including class-conditional ones.

The negative result is likelihood. The L_simple model's test bound was 3.75 bits per dimension, and the model trained on the true bound reached 3.70. Both beat published estimates for energy-based models and score matching but are not competitive with other likelihood-based models; the Sparse Transformer was at 2.80. A rate–distortion analysis explains why: more than half of the lossless codelength goes to imperceptible detail, which a model tuned for appearance has little reason to spend capacity on. On the 256×256 LSUN sets, sample quality landed in the range of ProgressiveGAN, behind the StyleGAN family.

The cost is stated plainly and it is severe: sampling requires many sequential network evaluations — on the order of a thousand in this paper — where a GAN generates in one. Diffusion traded a training problem for a sampling problem, and at publication that made it orders of magnitude slower to generate with. Rungs 5 and 7 are both attacks on this one sentence.

The objective is also acknowledged to be a reweighted version of the proper variational bound rather than the bound itself — the simplification that makes it a clean regression drops terms, and it is justified because it works better, not because it is exact. An honest and consequential piece of empirical pragmatism.

How it held up: it displaced rung 2 as the default for image generation within about two years, and the reasons are the ones above — stable training and full distribution coverage beat one-shot sampling speed, because speed turned out to be fixable and instability did not. Everything from rung 5 to rung 8 is a refinement of this rung, which makes it the most load-bearing paper on the ladder.

To generate, start from pure noise x_T ~ 𝒩(0, I). At each step, run the network to predict the noise, subtract off a scaled portion to get a cleaner estimate, add a touch of fresh noise for stochasticity, and repeat down to x₀. After all T steps you have a fresh image the model has effectively imagined from static.

DDPM produced image quality competitive with the best GANs of the time — without the instability or mode collapse — establishing diffusion as a serious, and soon dominant, family of generative models.

The obvious cost is speed. Generating one image means running the network hundreds or a thousand times — orders of magnitude slower than a GAN's single pass. DDPM also operates directly in pixel space, so high-resolution generation is very expensive in both compute and memory.

Both limits drove the follow-up work. DDIM and other fast samplers cut the step count dramatically; latent diffusion moved the whole process into a compressed space to slash the per-step cost. DDPM is the sturdy foundation those improvements were built on, not the final, efficient product.

DDPM is the paper that made diffusion work, and it launched the generation era we're living in. Every major image and video generator — DALL·E 2, Imagen, Midjourney, Stable Diffusion — descends from the forward/reverse framework and the noise-prediction objective defined here. When a model "denoises" its way to a picture, it's running DDPM's idea.

Its deeper lesson is about the power of the right reframing: a hard generative problem became a simple supervised one by choosing to predict noise. That move dethroned GANs as the default for high-fidelity generation and reoriented the whole field around iterative denoising.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Ten papers from the first latent-variable generator to video and real-time 3D. The diffusion half is strictly cumulative — each rung removes a cost the last one paid.

1. 1VAE2014
2. 2GANs2014
3. 3VQ-VAE2017
4. 4DDPM2020
5. 5DDIM2021
6. 6Classifier-Free Guidance2021
7. 7Latent Diffusion2022
8. 8Sora2024
9. 9NeRF2020
10. 103D Gaussian Splatting2023

All 110 papers, in 19 ladders →