Discrete
latents.

Rung 1's latent is a continuous vector, which makes the prior over it something you have to assume — the standard normal — rather than something you can model well. This rung's change is to make the latent discrete: the encoder's output at each spatial position is snapped to the nearest entry in a learned codebook, so an image becomes a grid of integer indices.

Why that is a large idea rather than a quantisation detail: a grid of indices is a sequence of tokens. Which means the prior over images can be modelled by exactly the machinery that models language — an autoregressive transformer over a vocabulary — instead of being assumed to be a Gaussian. The generative problem is split cleanly in two: learn a good tokeniser, then learn a strong sequence model over its tokens.

Three pieces make it work and each fixes a specific break. Choosing the nearest codeword is not differentiable, so the gradient arriving at the decoder's input is copied straight through to the encoder's output as if the snapping had not happened — a biased approximation that works. The codebook entries are pulled toward the encoder outputs assigned to them. And a commitment term pulls the encoder's outputs toward their chosen codewords, because without it the encoder can drift away from the codebook indefinitely and the quantisation error grows without bound.

A standard variational autoencoder maps each input to a continuous latent vector — a point in a smooth, real-valued space. That's flexible, but it's awkward to model: the strongest generative machinery we have — autoregressive Transformers — works over discrete tokens, a finite vocabulary, not a cloud of floats. VQ-VAE (van den Oord, Vinyals & Kavukcuoglu, 2017) closes that gap by making the latent itself discrete.

The move is simple to state: the encoder still outputs a continuous vector, but before it reaches the decoder it's quantized — replaced by the nearest entry in a small learned dictionary of vectors called the codebook. Each latent position becomes one of K possible symbols, an integer index. An image is no longer a fog of real numbers; it's a compact grid of discrete tokens, exactly the kind of thing a language model knows how to devour. The whole paper is about making that snap-to-nearest operation actually learnable.

The codebook is a learned set of K embedding vectors — the discrete vocabulary of the latent space. Think of it as a palette: a fixed number of prototype vectors that the model gets to place wherever they're most useful. When the encoder emits a continuous vector z_e(x), quantization measures its distance to every codebook entry and replaces it with the closest one, z_q. The index of that entry is the discrete token you store.

Both sides learn together. The encoder learns to produce vectors that land near useful codes; the codebook entries drift toward wherever the encoder's outputs actually cluster (in the original paper, the codebook is learned with a codebook-loss term that pulls each code toward the encoder vectors assigned to it; an exponential-moving-average update is the paper's described alternative). Over training, the K codes spread out to tile the space of representations the data needs — a self-organizing dictionary. Choose K and the grid size and you've fixed exactly how many bits the latent can hold: a hard, interpretable information bottleneck.

Quantization is a nearest-neighbor lookup: pick the codebook entry with the smallest squared distance to the encoder's vector.

The catch is the backward pass. The argmin — "which code was closest?" — is a hard, non-differentiable choice; there's no gradient telling the encoder how to nudge its output to change the winner smoothly. VQ-VAE sidesteps this with the straight-through estimator: on the way forward it uses the quantized code, but on the way back it pretends the quantization was the identity and copies the decoder's gradient directly onto the encoder's continuous output. The encoder gets trained as if the snap never happened. It's an approximation — but a remarkably effective one, and it's the trick that makes discrete latents learnable end-to-end.

Because the straight-through trick skips the quantizer on the backward pass, the encoder gets a reconstruction gradient but the codebook gets none — so VQ-VAE adds explicit terms to train the codes and to keep the encoder honest. The loss has three parts:

Left: reconstruction. Middle: the codebook loss moves the chosen code e toward the encoder output (sg = stop-gradient, so only e updates). Right: the commitment loss, weighted by β, pulls the encoder's output toward its code so the encoder "commits" and doesn't hop between codes — only z_e updates. The runnable version below computes the quantization and these losses.

The stop-gradient operator sg[·] is what lets one squared-distance be reused for two jobs: freeze the encoder side and the term trains the codebook; freeze the codebook side and (scaled by β) it trains the encoder to commit. The commitment weight β is the one real knob — too small and the encoder's outputs wander freely, growing faster than the codes can chase; the paper found results were robust across a broad range and used β = 0.25.

VQ-VAE, introduced in the paper Neural Discrete Representation Learning, is two models trained one after the other: an autoencoder with a quantized bottleneck, and an autoregressive prior over the codes it produces. While the autoencoder trains, the prior is a fixed uniform distribution; a strong prior is fitted only afterwards.

1. Encode to a grid of vectors. A convolutional encoder maps the input to a grid of D-dimensional vectors z_e(x). In the CIFAR-10 comparison it was two stride-2 convolutions with 4×4 windows followed by two residual blocks, 256 channels throughout. For 128×128 ImageNet images the grid was 32×32. For raw audio, six stride-2 convolutions made the latent sequence 64 times shorter than the waveform.
2. Quantize. Replace each grid vector by its nearest neighbor among K learned codebook vectors e₁ … e_K, and store the index. K = 512 in the image and speech-reconstruction experiments (the phoneme analysis below used 128 codes), so each position holds 9 bits.
3. Decode. A mirror-image decoder (residual blocks, then two stride-2 transposed convolutions) reconstructs x from the quantized grid z_q(x). For speech the decoder was a WaveNet-style dilated convolutional network conditioned on the codes and on a speaker identity.
4. Send gradients around the argmin. The reconstruction gradient at the decoder's input is copied straight onto z_e(x). The codebook gets its own term pulling each selected e toward the encoder outputs assigned to it, and a commitment term with β = 0.25 pulls those outputs toward their codes. Results were reported to change little for β from about 0.1 to 2.0. The paper also describes an alternative that updates each code as an exponential moving average of its assigned vectors.
5. Drop the KL. The posterior is one-hot and the training prior is uniform over K codes, so the KL term is the constant log K and is left out of the loss.
6. Fit the prior. Freeze the autoencoder, encode the training set into index grids, and train an autoregressive model on them: a PixelCNN for image grids, a WaveNet for audio sequences. Training the two stages jointly was left to future work.
7. Sample. Draw indices one position at a time from the prior, look up their codebook vectors and decode. For ImageNet the slow autoregressive model runs over 1,024 positions instead of the 49,152 subpixel values a pixel-level PixelCNN would generate one by one.

Why discrete codes resist posterior collapse

In a VAE the KL to the prior is a cost the encoder can drive to zero by ignoring the input, and a strong decoder lets it. VQ-VAE removes that lever. Every input is assigned some code, the KL is log K whichever code it is, and the code is the decoder's only source of information about x. Capacity is a hard count, K choices per position, rather than a soft penalty, which is why the paper could pair the codes with a powerful autoregressive WaveNet decoder and still have them used.

What the likelihood numbers mean

Because the decoder is only ever trained on z_q(x), the paper argues that once training converges it assigns essentially no probability to x under any other code. The log-likelihood is then approximately log p(x | z_q(x)) + log p(z_q(x)), a reconstruction term plus the prior's cost of the code, which is the kind of bound behind the bits-per-dimension figures below.

• q(z = k | x) — deterministic: all mass on one code, so the bottleneck adds no sampling noise and no estimator variance.
• log K — 9 bits per position for K = 512, a constant with no gradient, so it is dropped. For a whole grid it is the number of positions times log K, the bottleneck's exact capacity.
• sg[·] — stop-gradient: identity on the forward pass, zero derivative on the backward pass.
• −log p(x | zq) — trains the decoder and, through the copied gradient, the encoder; it never reaches the codebook.
• β — commitment weight, 0.25 in every experiment.

At the extremes: with β = 0 nothing ties the encoder to the codebook, and because the embedding space has no natural scale, its outputs can grow faster than the codes can follow. A codebook entry that no input selects gets zero from both of the last two terms, so it never moves, a dead code. And the copied gradient is exact only when z_q = z_e; the larger the quantization error, the more biased the encoder's update.

The paper is a set of demonstrations across images, audio and video more than a benchmark race, and its one head-to-head likelihood comparison is modest. On CIFAR-10 the authors trained a continuous VAE, VQ-VAE and VIMCO (a discrete-latent model trained with a multi-sample score-function estimator, here with 50 samples) using the same encoder and decoder, Adam at a learning rate of 2×10^⁻⁴, batch size 128, evaluated after 250,000 steps. All reported likelihoods are bounds.

The negative result is the first row. The discrete model is slightly worse than the continuous VAE on the likelihood bound, 4.67 against 4.51 bits per dimension, while clearly better than the previous discrete-latent method. Near parity is what the paper claims, and near parity is what the numbers support; the case for discreteness rests on what the codes make possible downstream, not on likelihood.

The other experiments show reach rather than scores. A code sequence 64 times shorter than the waveform cannot keep every detail of it, and the speech demonstrations emphasize content and speaker rather than waveform fidelity. On DeepMind Lab frames, a second VQ-VAE stacked on the first compressed the codes much further, and an action-conditional video model was trained entirely in the latent space.

The characteristic failure is codebook collapse: a subset of entries is never selected, receives no gradient, and stays where it was initialised, so the effective vocabulary is far smaller than the nominal one. Mitigations — reinitialising dead entries, using a moving average to update the codebook, normalising before matching — are standard practice and are all patches on the same dynamic.

The straight-through gradient is an acknowledged approximation, with no guarantee that it points in a helpful direction, and training is two-stage: the tokeniser and the prior are fitted separately, so the tokeniser cannot adapt to what the prior finds hard to model.

How it held up: it became the tokeniser, and that is a bigger role than generator. The reason phrases like "image tokens" and "audio tokens" are ordinary is this rung — discrete codes let images, video and speech be fed to sequence models built for text, which is the substrate under most multimodal generation and under the quantised targets on the multimodal ladder. Diffusion took the image-generation crown; discrete latents took everything that needed a shared interface with language.

VQ-VAE's real legacy is as a tokenizer for continuous data. Once an image is a small grid of discrete codes, you can hand those codes to a powerful separate model — an autoregressive Transformer or a diffusion prior — and generate in that compact token space instead of over millions of raw pixels. This "compress to tokens, then model the tokens" recipe is dramatically cheaper and produces sharper results, and it became the dominant pattern in generative vision.

The line of descent is direct: VQ-VAE-2 stacked hierarchical codes for high-resolution images; DALL·E tokenized images into a discrete grid and modeled them with a Transformer alongside text; VQGAN added a perceptual/adversarial loss for crisper codes; and the same idea powers discrete audio tokenizers behind modern speech and music models. Even latent diffusion — which uses a continuous latent — inherits the core insight that you should generate in a learned, compressed representation rather than pixel space. The specific trick of a nearest-neighbor codebook with a commitment loss turned out to be one of the most reused building blocks in generative AI.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Ten papers from the first latent-variable generator to video and real-time 3D. The diffusion half is strictly cumulative — each rung removes a cost the last one paid.

1. 1VAE2014
2. 2GANs2014
3. 3VQ-VAE2017
4. 4DDPM2020
5. 5DDIM2021
6. 6Classifier-Free Guidance2021
7. 7Latent Diffusion2022
8. 8Sora2024
9. 9NeRF2020
10. 103D Gaussian Splatting2023

All 110 papers, in 19 ladders →