VAE,
explained.

Rung 1, so the comparison is against the ordinary autoencoder. Train a network to compress an image to a small vector and reconstruct it, and it learns to do exactly that — and nothing more. The latent space it produces is full of holes: the encoder maps training images to scattered points, nothing constrains the space between them, and a randomly chosen latent decodes to noise. You have a compressor, not a generator, and the difference is that you cannot sample from it.

The change is to encode to a distribution instead of a point — a mean and a variance — and to add a penalty pulling each of those distributions toward a standard normal. Now the encoder is forced to spread its mass over a known, simple distribution rather than to hoard it in isolated spikes, so drawing a sample from that standard normal and decoding it lands somewhere the decoder has learned to handle. The objective is a reconstruction term traded against that penalty, which is a lower bound on the data's likelihood.

Two pieces of machinery make it trainable and both are worth carrying up the ladder. The penalty has a closed form for Gaussians, so it is an expression rather than an estimate. And sampling is not differentiable, which would break backpropagation — so instead of sampling from the encoder's distribution directly, you sample a fixed standard normal and shift and scale it by the encoder's outputs. The randomness moves to a leaf of the graph and the gradient flows through the arithmetic, which is a trick that reappears everywhere a network needs to sample.

An autoencoder is two networks: an encoder that compresses an input x into a small latent code z, and a decoder that rebuilds x from z. Train it to reconstruct and it learns a compact code. But as a generative model it fails: the encoder is free to scatter codes anywhere it likes, so the latent space becomes a lumpy archipelago — clusters of valid points separated by vast dead regions. Sample a random z and decode it, and you almost certainly land in a dead region and get nonsense.

The Variational Autoencoder (Kingma & Welling, 2013) fixes this by making the latent space well-behaved by construction. Two changes do it: the encoder outputs a distribution instead of a point, and a regularizer pulls every input's distribution toward a single simple prior — a standard normal. Now the space is smooth and gap-free, so a random draw from the prior decodes to something plausible. You've turned a compressor into a generator. The trick is doing this without breaking training — which is where the paper's two ideas come in.

Instead of emitting a single code, a VAE's encoder emits the parameters of a Gaussian: a mean μ and a variance σ² per latent dimension. The latent for an input is a cloud centered at μ with spread σ — a small region of the space that all decodes to (roughly) the same input. That alone starts to fill the gaps: nearby points now mean similar things, because each input claims a whole neighborhood, not a pinprick.

But clouds could still drift apart and leave holes. So the VAE adds the KL-divergence regularizer: a penalty for how far each encoder cloud q(z|x) = N(μ, σ²) sits from the shared prior p(z) = N(0, 1). This pressure herds all the clouds toward the origin and toward unit spread, so together they tile a single smooth blob you can sample from. The two forces are in tension — reconstruction wants each cloud tight and distinctive so it can rebuild its input; the KL wants every cloud to look like the same standard normal — and the VAE's whole behavior is that balance. Crank the KL up and the space is smooth but reconstructions blur; turn it down and you drift back toward a plain autoencoder.

There's a problem hiding in "sample z from N(μ, σ²)." Training needs gradients to flow back through the encoder to adjust μ and σ — but sampling is random, and you can't backpropagate through a random draw. The gradient path is cut.

The reparameterization trick rewrites the sample as z = μ + σ·ε, where ε is drawn from a fixed standard normal that carries no learnable parameters. Statistically this produces exactly the same distribution as sampling from N(μ, σ²) directly — but now the only randomness is ε, and μ and σ sit on a clean, differentiable path. Gradients flow through them as if the sampling weren't there. This one reframing is what turned the VAE from an elegant idea into a network you can train with ordinary backpropagation, and the trick reappears throughout modern generative modeling.

The true objective — maximize the data's log-likelihood log p(x) — is intractable. The VAE instead maximizes a tractable lower bound, the ELBO: a reconstruction term minus the KL from the encoder distribution to the prior.

Maximizing the ELBO = minimizing (reconstruction error + KL). The first term makes z carry enough about x to rebuild it; the second keeps q(z|x) near the standard-normal prior, so the latent space stays smooth and samplable.

For a Gaussian encoder and a standard-normal prior, the KL has a closed form — no sampling needed — which is what makes the whole thing cheap to optimize:

When μ = 0 and σ = 1 (the cloud exactly matches the prior), every term is 0 and the KL is 0 — no penalty. Push μ away from 0, or σ away from 1, and the penalty grows. The runnable version below computes this KL, the reparameterized sample, and the full VAE loss.

The paper is titled Auto-Encoding Variational Bayes, and the variational autoencoder is its worked example. The setting is general: a latent-variable model p_θ(x, z) = p_θ(z) p_θ(x | z) whose true posterior p_θ(z | x) cannot be computed, and a dataset too large for a per-example inference loop. The method, AEVB, learns the generative model and an approximate posterior together, one minibatch gradient step at a time.

1. Fix the generative model. The prior is p(z) = N(0, I). The decoder is an MLP that maps z to the parameters of p_θ(x | z): Bernoulli probabilities for MNIST pixels, or a Gaussian mean (squashed into (0, 1) by a sigmoid) for the real-valued Frey Face images.
2. Amortize inference with an encoder. Classical variational inference fits separate variational parameters for every training example. Here one MLP, the recognition model q_φ(z | x), reads x and outputs the mean μ and log-variance log σ^² of a diagonal Gaussian. Inference for a new image becomes one forward pass, and the cost of inference is paid once, in training the encoder.
3. Reparameterize the sample. Draw ε ~ N(0, I) and set z = μ + σ ⊙ ε. The paper points out that the same move works for any distribution with a tractable inverse CDF, any location-scale family, and compositions of these.
4. Estimate the bound. For each image in a minibatch of M = 100, draw L = 1 sample of z. Compute the KL to the prior in closed form and add log p_θ(x | z) at the sampled z. One sample per image is enough because the minibatch already averages over 100 of them.
5. Take one joint gradient step. Differentiate the estimate with respect to both θ and φ and update them together. The experiments used Adagrad, small random initial weights, and a light weight decay that amounts to a Gaussian prior on the parameters.
6. Generate. Sample z from the prior and decode. The pictures usually shown are the decoder's mean, not a draw from p_θ(x | z), which would add independent per-pixel noise.

Why a bound, and where it leaks

The quantity maximized is a lower bound on log p_θ(x), and the slack has a precise identity: the KL divergence from q_φ(z | x) to the true posterior. Raising the bound through φ shrinks that slack. Raising it through θ improves the model, but only as seen through the encoder, so if the true posterior is far from a diagonal Gaussian the bound stays loose and the decoder is trained under a weaker objective. Cremer, Li and Duvenaud (2018) later split this slack into an approximation gap, from the Gaussian family being too simple, and an amortization gap, from one shared encoder not finding the best Gaussian for each input.

The decoder likelihood sets the balance

The reconstruction term is a log-likelihood, so its scale depends on the likelihood you choose. A Bernoulli decoder gives binary cross-entropy. A Gaussian decoder with fixed variance σ_x^² gives squared error divided by 2σ_x^², which means that variance quietly sets how much reconstruction counts against the KL. Too large and the KL wins: samples blur and latent dimensions switch off. Too small and the model drifts back toward a plain autoencoder with holes in its latent space.

Two estimators, and one the paper rejects

The paper gives a generic estimator that samples log p_θ(x, z) − log q_φ(z | x) directly, usable when no closed-form KL exists, and a lower-variance one that computes the KL analytically, which is the VAE's. It rejects the standard score-function (REINFORCE) gradient because its variance is too high to be practical here.

• KL(q ‖ p(z | x)) — the gap. It is never negative, so 𝓛 ≤ log p(x), with equality only when the encoder's Gaussian is the true posterior.
• 𝔼[log p(x | z)] — expected reconstruction log-likelihood; its form is the decoder choice (Bernoulli, Gaussian).
• KL(q ‖ p(z)) — distance from the encoder's Gaussian to the prior, analytic here, which is why this estimator has lower variance than sampling it.
• z = μ + σ ⊙ ε — moves the randomness into ε so the gradient with respect to φ passes through μ and σ.

Because log p(x) does not involve φ, every gain in 𝓛 through φ is an equal loss of gap. At one extreme, a perfect encoder makes the bound tight and AEVB becomes maximum likelihood. At the other, a decoder that ignores z makes the true posterior equal the prior; an encoder that outputs the prior then pays zero KL and zero gap. The bound is tight and the latent carries nothing, which is why posterior collapse cannot be seen in the bound itself.

The experiments are small and were built to test the estimator, not to make good-looking pictures. MNIST digits got a Bernoulli decoder; the continuous Frey Face frames got a Gaussian decoder. Encoder and decoder were MLPs with one hidden layer each: 500 hidden units for MNIST and 200 for the much smaller Frey Face set, to limit overfitting. Minibatches of 100, one latent sample per image, Adagrad. The baselines were the wake-sleep algorithm, given the same encoder architecture, and, for a marginal-likelihood comparison in a very low-dimensional latent space, Monte Carlo EM with a Hybrid Monte Carlo sampler.

What the paper does not show matters as much. There are no sample-quality metrics (none of the later ones existed), the MNIST samples are visibly soft, and the likelihood figures are bounds or estimates. The follow-ups measured the gap the paper left open. IWAE (Burda, Grosse and Salakhutdinov, 2015) averaged k importance-weighted samples inside the logarithm, a bound that tightens as k grows, and its analysis found many VAE latent dimensions inactive. NVAE (Vahdat and Kautz, 2020) used a deep hierarchy of latent groups and careful normalization to produce sharp high-resolution faces, which weakened the view that blur is inherent to the variational bound.

The famous weakness is blurry samples, and the reason is structural rather than a tuning failure. The reconstruction term is typically a per-pixel likelihood, and when several outputs are plausible, the value that minimises an averaged per-pixel error is the average of them — which for images is a blur. The model is behaving correctly and the objective is asking for the wrong thing.

The second documented failure is what came to be called posterior collapse: when the decoder is powerful enough to model the data without help, the cheapest way to satisfy the penalty is to ignore the latent entirely, and the latent carries no information. The balance between the two terms is therefore delicate in a way the clean derivation does not suggest.

How it held up: as a generator of images it lost — to rung 2 on sharpness and then to rung 4 on everything. As a framework it is foundational, and its second career is the interesting part: the autoencoder described here, trained to compress an image into a compact latent, is exactly the component that makes rung 7 affordable. It stopped being the generative model and became the compressor the generative model runs inside, which is a better fate than the sharpness argument suggested.

The VAE's deepest legacy is the idea of a learned, well-structured latent space you can generate and interpolate in — and that idea powers systems far bigger than the original model. Most strikingly, latent diffusion (the basis of Stable Diffusion) runs its diffusion process not on raw pixels but inside a VAE-style latent space: an autoencoder compresses images to a compact latent, diffusion generates in that space, and the decoder paints the result back to pixels. The efficiency of modern image generation rests directly on that compression trick.

Beyond images, the reparameterization trick became a standard tool for any model that needs to backpropagate through a sampling step, and the ELBO framing connects deep learning to a whole tradition of variational inference. VAEs also seeded a family of successors — discrete-latent variants like VQ-VAE that swap the Gaussian latent for a learned codebook, powering later autoregressive image and audio models. The plain VAE rarely wins a sample-quality contest today, but its three gifts — an explicit latent, a likelihood-based objective, and differentiable sampling — are woven through the generative models that do.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Ten papers from the first latent-variable generator to video and real-time 3D. The diffusion half is strictly cumulative — each rung removes a cost the last one paid.

1. 1VAE2014
2. 2GANs2014
3. 3VQ-VAE2017
4. 4DDPM2020
5. 5DDIM2021
6. 6Classifier-Free Guidance2021
7. 7Latent Diffusion2022
8. 8Sora2024
9. 9NeRF2020
10. 103D Gaussian Splatting2023

All 110 papers, in 19 ladders →