Latent Diffusion,
explained.

Diffusion models generate by starting from pure noise and denoising it step by step until an image emerges. It works beautifully — but the earlier models did this directly on pixels. A 512×512 RGB image is nearly 800,000 numbers, and the model has to process all of them at every one of dozens or hundreds of denoising steps.

The cruel part is that most of those pixels carry redundant, high-frequency detail — texture and noise the eye barely registers. The model burns enormous compute perfecting bits that don't matter perceptually. That inefficiency put high-resolution diffusion out of reach for anyone without a fleet of GPUs, keeping the best image generators locked inside big labs.

The insight is to split the problem into two jobs done by two networks. First, a compressor learns to squeeze an image into a small latent representation that keeps the meaningful structure and throws away the imperceptible detail. Then the diffusion model does all its slow denoising work in that compact latent space — never touching raw pixels until the very end, when a decoder blows the finished latent back up to full resolution.

It's the difference between planning a painting as a rough thumbnail versus fussing over every brushstroke from the start. Get the composition right on the small sketch, cheaply, then render the detail once. The paper calls this a latent diffusion model — diffusion moved off the pixel grid and onto a tiny latent grid.

The compressor is a VAE (variational autoencoder), trained once, separately. Its encoder maps a 512×512 image down to roughly a 64×64 latent grid — about an 8× reduction per side, so tens of times fewer numbers to work with. Its decoder does the reverse, reconstructing a full image from a latent.

Crucially, the VAE is trained with perceptual and adversarial losses so it preserves what looks right rather than matching pixels exactly. It discards precisely the imperceptible high-frequency detail that was wasting the diffusion model's time. After training, the VAE is frozen and its latent space becomes the fixed "canvas" the diffusion model paints on.

Inside the latent space, it's ordinary diffusion. During training, a clean latent is progressively corrupted with noise; a U-Net is taught to predict and remove that noise one step at a time. At generation, you start from a random latent — pure noise — and run the U-Net repeatedly to denoise it into a coherent latent, which the VAE decoder then turns into an image.

The U-Net is the workhorse: an encoder-decoder with skip connections that's well suited to grid-shaped data. Because the grid it operates on is now tiny, each denoising step is cheap — the same number of steps as pixel diffusion, but each one a fraction of the cost. That's where the order-of-magnitude speedup comes from.

A generator you can't direct is a curiosity. Latent diffusion adds flexible conditioning through cross-attention. A prompt like "an astronaut riding a horse" is turned into embeddings by a text encoder, and at several layers inside the U-Net, the image latents attend to those text embeddings — asking, at every location, "which words should shape what I generate here?"

This is the same attention mechanism from Transformers, now bridging two modalities: image latents as queries, text tokens as keys and values. The general form means the same machinery can condition on text, another image, a segmentation map, or a layout — text-to-image is just the most famous case. It's how a sentence becomes a specific picture rather than a random one.

Latent diffusion matched or beat pixel-space diffusion on image quality benchmarks while using dramatically less compute for both training and generation. The efficiency was the story: high-resolution, text-conditioned image synthesis that a single consumer GPU could run in seconds.

Released publicly as Stable Diffusion, this architecture put a capable text-to-image model in millions of hands, sparked an explosion of tools, fine-tunes, and plugins, and turned "type a sentence, get a picture" into an everyday act rather than a research demo.

The compression is lossy. Because the VAE throws away detail, latent diffusion can fumble the finest textures and — notoriously — small structured things like text in a sign or the exact count of fingers. It's still an iterative sampler, so generation takes many steps unless sped up by later distillation methods.

And it inherits the ethics of its training data: models trained on huge web scrapes can reproduce biases, memorize and regurgitate copyrighted images, and enable convincing fakes. The very openness that made Stable Diffusion powerful also made these harms widely accessible — a tension the field is still working through.

Latent diffusion is the architecture that democratized generative imagery. By moving the expensive process into a compressed space, it turned a data-center capability into an open-source download — and the resulting ecosystem (Stable Diffusion, ControlNet, LoRA fine-tunes, and countless apps) reshaped art, design, and content overnight.

The pattern generalized far beyond images: "compress with an autoencoder, then generate in the latent space, conditioned by cross-attention" now underpins state-of-the-art video and audio generators too. When a modern model makes a video from a prompt, it is very likely running diffusion in a latent space — the idea this paper made standard.