Diffusion, and the
art of removing noise.

How do you turn random noise into a photorealistic image? Trying to produce it in one shot is impossibly hard. Diffusion's insight is to make the problem gradual and reversible: if you can learn to remove a little noise, you can start from pure noise and remove a little, then a little more, over many steps — and arrive at a clean image.

The training trick is to learn from the easy direction. Destroying an image by adding noise is trivial and well-defined; reversing that destruction is what's hard and useful. So diffusion models learn to undo a noising process they can generate perfect training examples for. Generation is just running that learned undo, step by step, starting from noise.

The forward (diffusion) process is the easy, hand-designed direction: take a real image and add a small amount of Gaussian noise repeatedly, over many timesteps, until after enough steps the image is indistinguishable from pure random noise. This process is fixed — no learning involved — and it's how you manufacture training data.

A crucial property makes this practical: you can jump to any noise level in one step (the math lets you add the equivalent of t steps of noise directly). So during training you can instantly produce "a clean image with exactly this much noise added," which is exactly the training example the network needs.

The reverse process is the hard, learned direction: go from noisier to less noisy. A neural network (typically a U-Net, or increasingly a transformer) is trained to take a noisy image and a timestep, and estimate how to remove a small increment of noise. Apply it repeatedly — noise → slightly cleaner → cleaner → … → image — and you've generated a sample.

Why step by step rather than one big jump? Because each small denoising step is a much easier, more accurate thing to learn than mapping pure noise directly to a photo. The gradualness is the whole point: many small, learnable corrections compose into an astonishingly hard generation task. The network never "draws" an image — it just keeps answering "what would this look like with a bit less noise?"

Here's the elegant part interviewers love. Training is a simple regression. Take a clean image, pick a random timestep t, add the corresponding known amount of Gaussian noise, and feed the noisy image (and t) to the network. Train it to predict the noise that was added, with a plain mean-squared-error loss between predicted and actual noise.

loss = ‖ ε − ε_θ(noisy image, t) ‖²

That's it — no adversarial game, no complex objective. Because you added the noise yourself, you have the exact ground-truth target for free. And knowing the noise lets you subtract it to denoise. This simplicity and stability — a straightforward supervised loss with unlimited self-generated training pairs — is a big reason diffusion overtook trickier generative approaches.

To sample (generate), start from pure random noise and run the reverse process: predict the noise, remove a bit, repeat for many steps until a clean image emerges. The number of steps is the central quality-vs-speed dial. The original formulation used many steps (hundreds or a thousand), which is slow; faster samplers (DDIM and successors) get high quality in far fewer steps (tens), and active research pushes toward single-digit or even one-step generation via distillation.

The tradeoff is intuitive: more steps mean smaller, more accurate corrections and usually better quality, but more compute and latency; fewer steps are faster but can lose fidelity. Sampling is where the per-image cost lives — which is why it's the knob every text-to-image service tunes for its latency and cost budget.

So far this generates a plausible image, not the one you asked for. Conditioning steers it: to make text-to-image, the model's denoising network is conditioned on a text embedding (from a text encoder), injected via cross-attention so that at every step the denoising is pulled toward matching the prompt. The same mechanism conditions on class labels, images (img2img), or masks (inpainting).

Classifier-free guidance controls how strongly the output obeys the prompt. The trick: run the model both with and without the conditioning, and amplify the difference by a guidance scale. A higher scale forces tighter prompt adherence (more literal, more saturated) at some cost to diversity and naturalness; a lower scale is more varied and natural but looser. That guidance slider is the knob behind "make it follow my prompt more."

Running diffusion directly on full-resolution pixels is enormously expensive — every one of many steps processes a huge image. Latent diffusion (the approach behind Stable Diffusion) fixes this: an autoencoder first compresses images into a much smaller latent representation, the diffusion process runs entirely in that compact latent space, and a decoder turns the final latent back into a full-resolution image.

Because the latent is far smaller than the pixel image, training and sampling become dramatically cheaper — which is precisely what made high-quality text-to-image runnable on consumer hardware. The diffusion math is identical; it just operates on compressed latents instead of raw pixels. Latent diffusion is the practical breakthrough that took diffusion from research to everywhere.

Diffusion didn't arrive first, but it won the image-generation crown. GANs (a generator fooling a discriminator) can be fast but are notoriously unstable to train and suffer mode collapse (limited diversity). VAEs are stable and give a nice latent space but tend to produce blurrier samples. Diffusion offers the best of both: stable, simple training (just predict noise) with state-of-the-art quality and diversity.

The price is speed — generation takes many network passes, not one — which is exactly what fast samplers, distillation, and latent diffusion attack. The trend is clear: diffusion's training stability and sample quality made it the dominant approach for images (and increasingly video and audio), with the ecosystem steadily closing the speed gap.