Latent Diffusion,
explained.

Rungs 4 to 6 denoise pixels, which means every one of the sampling steps is a full network pass at the image's full resolution. That is the remaining cost, and it is what kept high-resolution diffusion inside organisations with clusters.

The change rests on an observation about where the compute was going. Most of the bits in an image are perceptually irrelevant high-frequency detail — exact texture, sensor noise, the precise placement of individual leaves — and a diffusion model denoising pixels spends the majority of its capacity and its compute modelling them. But that detail is exactly what an autoencoder reconstructs well and cheaply. So: compress the image with an autoencoder into a much smaller latent, run the entire diffusion process there, and decode once at the end. The diffusion model now spends its capacity on semantics — what is in the picture and where — and the decoder handles texture.

Stated in units, that is an order-of-magnitude reduction in the compute of both training and sampling, which is the difference between a model that a lab runs and one that runs on a consumer graphics card. The second contribution is the conditioning mechanism: text is injected by cross-attention into the denoiser, which is general — anything you can encode as a sequence can steer generation the same way, and that is why the same architecture handles text, layouts, depth maps and image prompts without redesign. Note the ladder closing a loop here: rung 1, which lost as a generator, is the component that makes the winner affordable.

Diffusion models generate by starting from pure noise and denoising it step by step until an image emerges. It works beautifully — but the earlier models did this directly on pixels. A 512×512 RGB image is nearly 800,000 numbers, and the model has to process all of them at every one of dozens or hundreds of denoising steps.

The cruel part is that most of those pixels carry redundant, high-frequency detail — texture and noise the eye barely registers. The model burns enormous compute perfecting bits that don't matter perceptually. That inefficiency put high-resolution diffusion out of reach for anyone without a fleet of GPUs, keeping the best image generators locked inside big labs.

The insight is to split the problem into two jobs done by two networks. First, a compressor learns to squeeze an image into a small latent representation that keeps the meaningful structure and throws away the imperceptible detail. Then the diffusion model does all its slow denoising work in that compact latent space — never touching raw pixels until the very end, when a decoder blows the finished latent back up to full resolution.

It's the difference between planning a painting as a rough thumbnail versus fussing over every brushstroke from the start. Get the composition right on the small sketch, cheaply, then render the detail once. The paper calls this a latent diffusion model — diffusion moved off the pixel grid and onto a tiny latent grid.

The compressor is a VAE (variational autoencoder), trained once, separately. Its encoder maps a 512×512 image down to roughly a 64×64 latent grid — about an 8× reduction per side, so tens of times fewer numbers to work with. Its decoder does the reverse, reconstructing a full image from a latent.

Crucially, the VAE is trained with perceptual and adversarial losses so it preserves what looks right rather than matching pixels exactly. It discards precisely the imperceptible high-frequency detail that was wasting the diffusion model's time. After training, the VAE is frozen and its latent space becomes the fixed "canvas" the diffusion model paints on.

Inside the latent space, it's ordinary diffusion. During training, a clean latent is progressively corrupted with noise; a U-Net is taught to predict and remove that noise one step at a time. At generation, you start from a random latent — pure noise — and run the U-Net repeatedly to denoise it into a coherent latent, which the VAE decoder then turns into an image.

The U-Net is the workhorse: an encoder-decoder with skip connections that's well suited to grid-shaped data. Because the grid it operates on is now tiny, each denoising step is cheap — the same number of steps as pixel diffusion, but each one a fraction of the cost. That's where the order-of-magnitude speedup comes from.

And "a fraction" is a precise fraction. A downsample factor f shrinks each side, so the number of spatial positions the U-Net processes drops by f² — and the U-Net's attention layers, being quadratic in positions, drop by f⁴:

Same denoising steps, each on a 64×-smaller canvas — the entire efficiency story, in exponents of one number.

A generator you can't direct is a curiosity. Latent diffusion adds flexible conditioning through cross-attention. A prompt like "an astronaut riding a horse" is turned into embeddings by a text encoder, and at several layers inside the U-Net, the image latents attend to those text embeddings — asking, at every location, "which words should shape what I generate here?"

This is the same attention mechanism from Transformers, now bridging two modalities: image latents as queries, text tokens as keys and values. The general form means the same machinery can condition on text, another image, a segmentation map, or a layout — text-to-image is just the most famous case. It's how a sentence becomes a specific picture rather than a random one.

Latent diffusion matched or beat pixel-space diffusion on image quality benchmarks while using dramatically less compute for both training and generation. The efficiency was the story: high-resolution, text-conditioned image synthesis that a single consumer GPU could run in seconds.

Released publicly as Stable Diffusion, this architecture put a capable text-to-image model in millions of hands, sparked an explosion of tools, fine-tunes, and plugins, and turned "type a sentence, get a picture" into an everyday act rather than a research demo.

The compression is lossy. Because the VAE throws away detail, latent diffusion can fumble the finest textures and — notoriously — small structured things like text in a sign or the exact count of fingers. It's still an iterative sampler, so generation takes many steps unless sped up by later distillation methods.

And it inherits the ethics of its training data: models trained on huge web scrapes can reproduce biases, memorize and regurgitate copyrighted images, and enable convincing fakes. The very openness that made Stable Diffusion powerful also made these harms widely accessible — a tension the field is still working through.

Latent diffusion is the architecture that democratized generative imagery. By moving the expensive process into a compressed space, it turned a data-center capability into an open-source download — and the resulting ecosystem (Stable Diffusion, ControlNet, LoRA fine-tunes, and countless apps) reshaped art, design, and content overnight.

The pattern generalized far beyond images: "compress with an autoencoder, then generate in the latent space, conditioned by cross-attention" now underpins state-of-the-art video and audio generators too. When a modern model makes a video from a prompt, it is very likely running diffusion in a latent space — the idea this paper made standard.

The autoencoder is a hard ceiling, and this is the honest cost: anything the encoder discards, the diffusion model cannot generate, because it never sees it and the decoder must invent it. That is precisely why models of this family were for years bad at small text inside images and at fine high-frequency structure — the failure is in the compressor, not the generator, and no amount of diffusion capacity fixes it. The compression ratio is the single most consequential design choice in the system.

Training is also two-stage, so the autoencoder is fitted without knowing what the diffusion model will find difficult, and an adversarial term is used in the decoder to keep reconstructions sharp — which quietly means rung 2 is in the pipeline too.

How it held up: this is the architecture of deployed image generation. Its release as open weights is also the single largest reason a public ecosystem of fine-tunes, adapters and tooling exists for image models at all, which is a social consequence of a compute decision. The general lesson transfers past images: run the expensive iterative process in the smallest representation that preserves what you care about, and let a cheap decoder restore the rest.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

Ten papers from the first latent-variable generator to video and real-time 3D. The diffusion half is strictly cumulative — each rung removes a cost the last one paid.

1. 1VAE2014
2. 2GANs2014
3. 3VQ-VAE2017
4. 4DDPM2020
5. 5DDIM2021
6. 6Classifier-Free Guidance2021
7. 7Latent Diffusion2022
8. 8Sora2024
9. 9NeRF2020
10. 103D Gaussian Splatting2023

All 110 papers, in 19 ladders →