A generative model — a GAN, a VAE, the decoder of a diffusion model — is a function that takes a short vector of numbers, the latent codez, and turns it into a full image. The astonishing part isn’t that it can produce an image from numbers; it’s that the space of those numbers is organized. Training doesn’t just teach the model to memorize outputs — it arranges the latent space so that its geometry means something. Three properties fall out, and they’re what make generative models feel magical. First, continuity: codes that are close together produce images that look similar, so there are no jarring jumps — the space is smooth. Second, interpolation: walk in a straight line from one code to another and the image morphs smoothly between the two, passing through plausible in-between images the model was never explicitly shown. Third, semantic directions: particular directions in the space correspond to meaningful attributes — add a certain vector and a face gains a smile, or an object rotates — so you can steer a single property while leaving the rest intact. Below is a small, hand-built generator so you can see these properties directly: move its latent code, interpolate between two points, and push along a semantic direction.
The input vector to a generator. Every point in this space decodes to an image.
continuity
Nearby latent codes produce similar images — the space is smooth, with no sudden jumps.
interpolation
Walking a straight line between two codes morphs one image smoothly into the other.
semantic direction
A direction in latent space that changes one meaningful attribute (e.g. expression) while leaving others fixed.
generator.js — a code becomes a picture
Ready
On the left is a 2-D latent space; on the right is what a toy generator decodes from the current code. Move the code with the arrows and watch the image change smoothly (continuity). Interpolate A→B to morph between two codes. Then walk the expression direction to steer one attribute.
-0.6
z₁ · expression
-0.6
z₂ · energy
sleepy frown
decoded
How it works
Why should a pile of numbers end up meaningfully organized? Because of the pressure training puts on the generator. A generator is trained to map latent codes drawn from a simple, smooth distribution (usually a Gaussian) onto the complicated distribution of real images, and to fool a discriminator (in a GAN) or reconstruct data (in a VAE). To succeed with a continuous input distribution, the mapping it learns must itself be continuous — small changes in z can’t cause wild jumps, or the model couldn’t consistently produce realistic images across the whole input distribution. That continuity is exactly what makes interpolation work: the straight-line path between two codes is made of nearby points, each decoding to a plausible image, so you glide smoothly from one output to the other through valid in-betweens the model synthesizes on the fly (this is the famous “latent walk,” and it’s strong evidence a model has learned genuine structure rather than memorized samples). Semantic directions emerge because the model discovers that varying certain factors of the data — pose, lighting, the presence of a smile — is an efficient way to explain the variation in real images, so those factors get encoded as (often roughly linear) directions in latent space; you can find a “smile vector” by averaging the codes of smiling faces and subtracting the codes of non-smiling ones, then add it to any code to add a smile. Real generators differ from this toy in scale and messiness — the space is hundreds of dimensions, directions are entangled rather than perfectly clean, and techniques like StyleGAN’s intermediate latent space or diffusion’s noise schedule exist partly to make the geometry more controllable — but the three properties you can feel here (continuity, interpolation, steerable directions) are precisely why we say a generative model has learned a representation, not just a lookup table. It’s also the foundation of practical image editing: to change one attribute of a generated (or inverted real) image, you find its code and move along the corresponding direction.
1
A code decodes to an image
The generator is a function from a latent vector z to an image. Every point in the latent space corresponds to some output — the space is the model’s vocabulary of things it can make.
2
Continuity: near codes, similar images
Nudge z a little and the image changes a little. The learned mapping is smooth, because the generator was trained to map a continuous input distribution onto realistic images — there are no sudden jumps.
3
Interpolation morphs between codes
Walk a straight line from code A to code B and the image morphs smoothly through plausible in-betweens the model synthesizes on the fly — evidence it learned real structure, not a lookup table of memorized samples.
✓
Directions carry meaning
Specific directions correspond to attributes: move along the “expression” axis and only the expression changes. Find such a direction (e.g. average smiling minus non-smiling codes) and you can steer one property of any image — the basis of latent-space editing.
Generator
z (vector) → image
Continuity
near codes → similar images
Interpolation
straight line morphs
Direction
a steerable attribute
The code
# A generator is just a function z -> image
image = G(z) # z is a short latent vector# Continuity: nearby z -> similar images (smooth mapping)# Interpolation: walk a straight line between two codesfor t in linspace(0, 1, 30):
z_t = (1 - t) * z_A + t * z_B # morphs A into B, frame by frame
show(G(z_t))
# Semantic direction: find an attribute vector, then steer it
smile_dir = mean(z_smiling) - mean(z_not_smiling)
edited = G(z + alpha * smile_dir) # add a smile, keep the rest
Quick check
1. What does the latent code (z) represent for a generative model?
A generator is a function from a latent vector z to an image. The latent space is the set of all such vectors, and every point in it decodes to some output — so the space is effectively the model’s vocabulary of things it can generate.
2. What does it mean that latent space is continuous, and why does it matter?
Continuity means the mapping from z to image is smooth: a small step in latent space yields a small change in the image, with no jarring jumps. This is exactly why walking a straight line between two codes morphs one image smoothly into the other through plausible in-betweens.
3. What is a semantic direction in latent space?
Training tends to encode meaningful factors of variation as directions in latent space. Moving along such a direction changes a single attribute — add a “smile vector” and a face smiles — which is the basis of latent-space image editing. In real models these directions are somewhat entangled rather than perfectly clean.
FAQ
What is latent space in a generative model?
The space of input vectors (latent codes) a generative model decodes into outputs like images. Rather than storing pictures, the model learns a function turning a short vector into a full image, and the set of all those vectors is the latent space. Training organizes it: nearby codes make similar images (continuity), straight-line paths morph smoothly (interpolation), and particular directions mean attributes (semantic directions). So it acts as a compact, structured representation of everything the model can generate.
Why does interpolating between two latent codes create a smooth morph?
Because the generator’s mapping from code to image is continuous. Walking a straight line from A to B passes through codes near their neighbors, and continuity means each decodes to a similar image — so outputs change gradually, gliding from A’s image to B’s through plausible intermediates the model synthesizes on the fly. This smooth interpolation is often taken as evidence the model learned real structure (a representation) rather than memorizing training examples, since the in-between images are coherent things it was never shown.
How do you find and use a semantic direction (like a "smile vector")?
Collect latent codes for examples with an attribute and codes for examples without it, then take the difference of their averages: mean(smiling) − mean(not-smiling) gives an approximate "smile direction." Adding a scaled version to any code makes the image smile more; subtracting it, less — ideally without changing other features. In practice directions are entangled (pushing "smile" may nudge age), so techniques like StyleGAN’s style space or unsupervised direction discovery aim for cleaner, disentangled directions. This underlies latent-space image editing: find an image’s code and move along the attribute direction.
What is the difference between how GANs, VAEs, and diffusion models use latent space?
All decode from a latent representation, but differently. A GAN maps random latents to images by fooling a discriminator — smooth space, great for interpolation, but finding the code for a real image needs a separate inversion step. A VAE learns an encoder and decoder together, so it can map images to codes and back, with a latent regularized to be smooth (often at some cost to sharpness). Diffusion models generate by denoising random noise, so the "latent" is the noise plus trajectory, and control comes from steering the denoising (and, in latent diffusion, a learned compressed space). The shared idea is a structured, navigable representation.