VAE,
explained.

An autoencoder is two networks: an encoder that compresses an input x into a small latent code z, and a decoder that rebuilds x from z. Train it to reconstruct and it learns a compact code. But as a generative model it fails: the encoder is free to scatter codes anywhere it likes, so the latent space becomes a lumpy archipelago — clusters of valid points separated by vast dead regions. Sample a random z and decode it, and you almost certainly land in a dead region and get nonsense.

The Variational Autoencoder (Kingma & Welling, 2013) fixes this by making the latent space well-behaved by construction. Two changes do it: the encoder outputs a distribution instead of a point, and a regularizer pulls every input's distribution toward a single simple prior — a standard normal. Now the space is smooth and gap-free, so a random draw from the prior decodes to something plausible. You've turned a compressor into a generator. The trick is doing this without breaking training — which is where the paper's two ideas come in.

Instead of emitting a single code, a VAE's encoder emits the parameters of a Gaussian: a mean μ and a variance σ² per latent dimension. The latent for an input is a cloud centered at μ with spread σ — a small region of the space that all decodes to (roughly) the same input. That alone starts to fill the gaps: nearby points now mean similar things, because each input claims a whole neighborhood, not a pinprick.

But clouds could still drift apart and leave holes. So the VAE adds the KL-divergence regularizer: a penalty for how far each encoder cloud q(z|x) = N(μ, σ²) sits from the shared prior p(z) = N(0, 1). This pressure herds all the clouds toward the origin and toward unit spread, so together they tile a single smooth blob you can sample from. The two forces are in tension — reconstruction wants each cloud tight and distinctive so it can rebuild its input; the KL wants every cloud to look like the same standard normal — and the VAE's whole behavior is that balance. Crank the KL up and the space is smooth but reconstructions blur; turn it down and you drift back toward a plain autoencoder.

There's a problem hiding in "sample z from N(μ, σ²)." Training needs gradients to flow back through the encoder to adjust μ and σ — but sampling is random, and you can't backpropagate through a random draw. The gradient path is cut.

The reparameterization trick rewrites the sample as z = μ + σ·ε, where ε is drawn from a fixed standard normal that carries no learnable parameters. Statistically this produces exactly the same distribution as sampling from N(μ, σ²) directly — but now the only randomness is ε, and μ and σ sit on a clean, differentiable path. Gradients flow through them as if the sampling weren't there. This one reframing is what turned the VAE from an elegant idea into a network you can train with ordinary backpropagation, and the trick reappears throughout modern generative modeling.

The true objective — maximize the data's log-likelihood log p(x) — is intractable. The VAE instead maximizes a tractable lower bound, the ELBO: a reconstruction term minus the KL from the encoder distribution to the prior.

Maximizing the ELBO = minimizing (reconstruction error + KL). The first term makes z carry enough about x to rebuild it; the second keeps q(z|x) near the standard-normal prior, so the latent space stays smooth and samplable.

For a Gaussian encoder and a standard-normal prior, the KL has a closed form — no sampling needed — which is what makes the whole thing cheap to optimize:

When μ = 0 and σ = 1 (the cloud exactly matches the prior), every term is 0 and the KL is 0 — no penalty. Push μ away from 0, or σ away from 1, and the penalty grows. The runnable version below computes this KL, the reparameterized sample, and the full VAE loss.

The VAE gave deep learning a principled, trainable generative model with an explicit latent space:

The trade-off, well known even at the time, is that VAE samples tend to be blurry — the Gaussian assumptions and the averaging pressure of the reconstruction term smooth out fine detail. That softness is why sharp image synthesis later leaned on GANs and then diffusion. But the VAE's structured latent space and stable training kept it central, especially as a component rather than a standalone image generator.

The VAE's deepest legacy is the idea of a learned, well-structured latent space you can generate and interpolate in — and that idea powers systems far bigger than the original model. Most strikingly, latent diffusion (the basis of Stable Diffusion) runs its diffusion process not on raw pixels but inside a VAE-style latent space: an autoencoder compresses images to a compact latent, diffusion generates in that space, and the decoder paints the result back to pixels. The efficiency of modern image generation rests directly on that compression trick.

Beyond images, the reparameterization trick became a standard tool for any model that needs to backpropagate through a sampling step, and the ELBO framing connects deep learning to a whole tradition of variational inference. VAEs also seeded a family of successors — masked and discrete-latent variants like VQ-VAE that swap the Gaussian latent for a learned codebook, powering later autoregressive image and audio models. The plain VAE rarely wins a sample-quality contest today, but its three gifts — an explicit latent, a likelihood-based objective, and differentiable sampling — are woven through the generative models that do.