Discrete
latents.

A standard variational autoencoder maps each input to a continuous latent vector — a point in a smooth, real-valued space. That's flexible, but it's awkward to model: the strongest generative machinery we have — autoregressive Transformers — works over discrete tokens, a finite vocabulary, not a cloud of floats. VQ-VAE (van den Oord, Vinyals & Kavukcuoglu, 2017) closes that gap by making the latent itself discrete.

The move is simple to state: the encoder still outputs a continuous vector, but before it reaches the decoder it's quantized — replaced by the nearest entry in a small learned dictionary of vectors called the codebook. Each latent position becomes one of K possible symbols, an integer index. An image is no longer a fog of real numbers; it's a compact grid of discrete tokens, exactly the kind of thing a language model knows how to devour. The whole paper is about making that snap-to-nearest operation actually learnable.

The codebook is a learned set of K embedding vectors — the discrete vocabulary of the latent space. Think of it as a palette: a fixed number of prototype vectors that the model gets to place wherever they're most useful. When the encoder emits a continuous vector z_e(x), quantization measures its distance to every codebook entry and replaces it with the closest one, z_q. The index of that entry is the discrete token you store.

Both sides learn together. The encoder learns to produce vectors that land near useful codes; the codebook entries drift toward wherever the encoder's outputs actually cluster (in the original paper, the codebook is updated to track the mean of the encoder vectors assigned to it). Over training, the K codes spread out to tile the space of representations the data needs — a self-organizing dictionary. Choose K and the grid size and you've fixed exactly how many bits the latent can hold: a hard, interpretable information bottleneck.

Quantization is a nearest-neighbor lookup: pick the codebook entry with the smallest squared distance to the encoder's vector.

The catch is the backward pass. The argmin — "which code was closest?" — is a hard, non-differentiable choice; there's no gradient telling the encoder how to nudge its output to change the winner smoothly. VQ-VAE sidesteps this with the straight-through estimator: on the way forward it uses the quantized code, but on the way back it pretends the quantization was the identity and copies the decoder's gradient directly onto the encoder's continuous output. The encoder gets trained as if the snap never happened. It's an approximation — but a remarkably effective one, and it's the trick that makes discrete latents learnable end-to-end.

Because the straight-through trick skips the quantizer on the backward pass, the encoder gets a reconstruction gradient but the codebook gets none — so VQ-VAE adds explicit terms to train the codes and to keep the encoder honest. The loss has three parts:

Left: reconstruction. Middle: the codebook loss moves the chosen code e toward the encoder output (sg = stop-gradient, so only e updates). Right: the commitment loss, weighted by β, pulls the encoder's output toward its code so the encoder "commits" and doesn't hop between codes — only z_e updates. The runnable version below computes the quantization and these losses.

The stop-gradient operator sg[·] is what lets one squared-distance be reused for two jobs: freeze the encoder side and the term trains the codebook; freeze the codebook side and (scaled by β) it trains the encoder to commit. The commitment weight β is the one real knob — too small and the encoder's outputs wander freely, growing faster than the codes can chase; the paper found results were robust across a broad range and used β = 0.25.

VQ-VAE demonstrated that a discrete latent can match continuous ones on reconstruction while being far more useful downstream:

It isn't free: the straight-through estimator is a biased gradient approximation, codebook usage can collapse to a few entries if unbalanced, and the reconstruction ceiling is set by K and the grid — a hard bottleneck. But the central result was decisive: you can compress continuous data into discrete tokens and lose little, which opens the door to modeling those tokens with the full power of sequence models.

VQ-VAE's real legacy is as a tokenizer for continuous data. Once an image is a small grid of discrete codes, you can hand those codes to a powerful separate model — an autoregressive Transformer or a diffusion prior — and generate in that compact token space instead of over millions of raw pixels. This "compress to tokens, then model the tokens" recipe is dramatically cheaper and produces sharper results, and it became the dominant pattern in generative vision.

The line of descent is direct: VQ-VAE-2 stacked hierarchical codes for high-resolution images; DALL·E tokenized images into a discrete grid and modeled them with a Transformer alongside text; VQGAN added a perceptual/adversarial loss for crisper codes; and the same idea powers discrete audio tokenizers behind modern speech and music models. Even latent diffusion — which uses a continuous latent — inherits the core insight that you should generate in a learned, compressed representation rather than pixel space. The specific trick of a nearest-neighbor codebook with a commitment loss turned out to be one of the most reused building blocks in generative AI.