V-JEPA,
explained.

Masked autoencoding on the vision ladder hides part of the input and reconstructs its pixels. For video that objective is asking for something impossible and this rung says so: much of what happens in the next frames is genuinely unpredictable — the exact shape of a flame, the precise position of every leaf, the individual grain of a texture. A model forced to predict those pixels spends its capacity modelling noise, and is penalised for not knowing what cannot be known.

The change is to move the prediction into representation space. Instead of reconstructing the hidden region's pixels, predict the features that an encoder produces for that region. Because those features are themselves learned, unpredictable detail can simply be dropped from them — the representation is free to encode "flame here, moving upward" and not the particular shape, and the prediction task becomes solvable without demanding clairvoyance. Abstraction is not a side effect; it is what the objective permits.

That freedom creates the obvious risk: if the encoder may discard anything, the cheapest solution is to discard everything and predict a constant. The prevention is the same device as self-distillation on the vision ladder — the target features come from a slowly-updated moving average of the encoder being trained, so the target is stable enough to be a signal and moves with the model rather than being a fixed thing to collapse onto. Stated in units, the payoff is where masked reconstruction is weak: strong performance from frozen features on video understanding, with no fine-tuning.

Self-supervised learning turns unlabeled data into a teacher by hiding part of the input and asking the model to fill it in. For text, the hidden thing is a token and the task is well-posed. For video, the naive version — hide some pixels, predict the pixels — runs into trouble: a huge fraction of what's under the mask is inherently unpredictable. The exact texture of grass, the grain of a shadow, sensor noise — no model can or should nail those, yet a pixel loss demands it try.

V-JEPA — Video Joint-Embedding Predictive Architecture — reframes the task. Instead of predicting the masked pixels, it predicts their representation: the features an encoder produces for that region. The prediction happens entirely in latent space. The point isn't to redraw the video; it's to know what should be there at a level of abstraction where the answer is actually predictable. That single shift — from reconstruction to representation prediction — is the whole idea, and it comes straight from Yann LeCun's argument that world models should predict abstractly, not pixel-by-pixel.

To see the problem concretely, split any patch into two parts: predictable structure (there's a face here, a hand moving left) and unpredictable detail (the exact micro-texture, the noise). A masked autoencoder reconstructs pixels, so its loss is measured against structure and detail. The best it can do on the detail is predict the average and eat the variance — an irreducible error floor it can never cross, no matter how much capacity you throw at it.

Worse, chasing that floor is actively harmful: gradient after gradient nudges the model to memorize surface statistics that don't transfer to recognizing actions or objects. The capacity spent modeling noise is capacity not spent on meaning. Pixel reconstruction isn't wrong, exactly — it just optimizes partly the wrong target. What you want is a loss that stops caring about the unpredictable part. That's exactly what moving the target into representation space does.

V-JEPA has three pieces: a context encoder that sees the visible part of the video, a predictor that guesses the masked region's features from that context, and a target encoder that produces the features being predicted. The loss is simply the distance between predicted features and target features — in latent space, no pixels involved.

But predicting in latent space has an obvious cheat: if the encoder maps everything to the same constant vector, the predictor's job is trivial and the loss goes to zero — while the model learns nothing. This is representation collapse, and it's the central risk of every joint-embedding method. V-JEPA blocks it with asymmetry: the target encoder isn't trained directly; it's an exponential moving average of the context encoder, with a stop-gradient so no gradient flows into the target. That slowly-drifting, non-differentiable target makes the constant solution unstable, so the encoder is pushed to produce features that are actually informative.

Model a patch as predictable structure s plus zero-mean unpredictable noise. Reconstructing pixels, the best predictor guesses the mean, and its error is the noise variance it cannot remove:

No matter the model, a pixel loss pays for every bit of unpredictable detail. More noise → strictly worse achievable error.

Now predict a feature instead. Let the encoder abstract the patch — say, average away the zero-mean noise — so its target is (near) the clean structure. A predictor that knows the structure hits it exactly:

The encoder threw away the unpredictable part before it became a target, so the loss never asks the model to predict noise. Adding more noise worsens the pixel target but leaves the feature target unchanged — the whole point. The runnable version below makes both errors concrete and shows the feature target staying invariant as noise grows.

V-JEPA trains a video encoder with a single regression loss: the paper's models use no pretrained image encoder, no text, no negative examples and no pixel reconstruction. Three networks take part in training, but only two of them receive gradients.

1. Sample a clip. 16 frames, taking every fourth frame, which covers about two seconds of video, at 224 × 224 pixels; the largest model is also trained at 384.
2. Cut it into tubelets. Each token covers 2 frames × 16 × 16 pixels, giving 8 × 14 × 14 = 1,568 tokens per clip.
3. Mask with multi-block tubes. Two kinds of mask are sampled: short-range, the union of 8 random blocks each covering about 15% of a frame, and long-range, the union of 2 blocks each covering about 70%, with aspect ratios between 0.75 and 1.5. Each spatial mask is repeated through every frame, so a hidden region stays hidden at all time steps, which the paper says limits information leaking between neighboring frames. On average about 90% of tokens are hidden.
4. Encode the context. The encoder, a ViT-L/16 or ViT-H/16, processes only the roughly 10% of tokens left visible.
5. Predict the hidden features. The predictor, a narrow Transformer of 12 blocks with width 384, receives the encoder's outputs plus one learnable mask token per hidden position, each carrying that position's embedding, and outputs a feature vector for every hidden position.
6. Compute the targets. A target encoder, an exponential moving average of the encoder's weights, processes the whole unmasked clip; its outputs at the hidden positions are the targets, and no gradient flows into it.
7. Update. The loss is the L1 distance between predictions and targets over the hidden tokens; the authors report that L1 trained more stably. Gradients update the encoder and the predictor, then the target encoder moves toward the encoder, with an averaging momentum that rises on a schedule from 0.998 toward 1.0 over training.
8. Data. VideoMix2M: about 2 million videos drawn from HowTo100M, Kinetics-400, -600 and -700, and Something-Something-v2, with overlap with their validation sets removed.

How the features are judged

The encoder is frozen and an attentive probe is trained on top for each task: a learnable query cross-attends over the encoder's output tokens, followed by a small MLP and a linear classifier. The tasks span motion-heavy video (Something-Something-v2), appearance-heavy video (Kinetics-400), action localization (AVA) and still images (ImageNet-1K, Places205, iNaturalist 2021).

To see what the features contain, the authors trained a separate conditional diffusion decoder, with the encoder and predictor frozen, that turns the predicted features for a hidden region into pixels. The decoded predictions are coherent in space and time with the visible parts of the clip, which is evidence that the features carry the hidden content's layout and motion, inspected through a second model rather than read off directly.

• x — the visible tubelets, about 10% of 1,568
• y — the hidden tubelets; Δy — learnable mask tokens carrying the positions of y
• Eθ — the encoder, the network kept afterwards; Pφ — the predictor
• Ēθ — the target encoder, a running average of past E_θ weights, run on the full clip; Ē_θ(y) denotes its outputs at the hidden positions
• sg — stop-gradient; ‖·‖1 — L1 distance, averaged over hidden tokens
• m — the averaging momentum

At the extremes: if E_θ mapped every input to one constant vector, the target would be that constant too and the loss would be exactly zero, a collapsed solution that learns nothing. Nothing in the equation forbids it; the stop-gradient and the slowly averaged target make it a solution that training does not reach in practice. With m = 1 from the start, the target would stay at its random initialization and the encoder would be regressing onto random features; as m approaches 1 late in training, the target simply settles. Using L1 rather than squared error means a few large target values pull the prediction less.

ViT-L/16 and ViT-H/16 encoders, the latter also at 384 pixels, were pre-trained on VideoMix2M and evaluated frozen with attentive probes. The baselines were video models trained by pixel reconstruction (VideoMAE, OmniMAE, Hiera) and image models (DINOv2, OpenCLIP, I-JEPA).

When the labels for training the probe were cut roughly tenfold, V-JEPA lost less accuracy than the pixel-reconstruction baselines, so its lead widened. Under full fine-tuning its ViT-L was the best model of that size and matched Hiera-L on Something-Something-v2, while having seen far fewer samples during pre-training.

The split with image models is the informative negative result. Appearance-heavy tasks such as Kinetics-400 and ImageNet still favor a model trained on a vast, diverse image collection, while the benchmark whose labels depend on motion, such as pushing something versus pulling it, favors the model that learned by predicting hidden regions across time.

The honest difficulty is that the target is learned, so what counts as a good prediction is partly defined by the thing being trained. That is what makes the objective tractable and it also means there is no built-in reconstruction to look at (the authors train a separate decoder afterwards just to visualize what the predictions encode) to check that the representation kept what matters — with masked autoencoding you can inspect the output and see what the model understood, and here you cannot. Evaluation is necessarily indirect, through downstream tasks.

The anti-collapse machinery is a balance rather than a guarantee, with the same caveat as on the vision ladder: it works reliably and there is no argument that it must.

How it held up: joint-embedding prediction became a major line of self-supervised work, and the intellectual argument is the durable part — do not ask a model to predict what is unpredictable. That reframes a large amount of generative and world-model work, including the tension visible at rung 8 of the generative ladder: a model trained to produce plausible pixels learns what video looks like, and a model trained to predict representations is at least aimed at what the world does. Which of those is the better route to understanding is exactly the open question, and this rung is the clearest statement of one side.

V-JEPA sits inside a larger thesis: that the path to machines which understand the world runs through predicting abstract representations of the future, not generating pixels of it. It contrasts sharply with generative video models like Sora, which do synthesize pixels. Both learn about the world from video, but their bets differ — generate the world, or model it in the abstract. JEPA is the non-generative bet: don't paint the future, predict what it will be like.

The family is broad — image JEPA (I-JEPA) came first, and the joint-embedding-plus-anti-collapse pattern connects to self-distillation methods that also learn without reconstructing pixels. What V-JEPA adds is the temporal dimension and a clean demonstration that, for video, abstraction beats reconstruction. Whether representation prediction or pixel generation is the better road to world models is still open — but V-JEPA made the abstraction road concrete, measurable, and hard to dismiss.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Joining text to images, audio, video and finally to a robot arm. Starts with the contrastive objective the whole field standardised on.

1. 1CLIP2021
2. 2Flamingo2022
3. 3LLaVA2023
4. 4wav2vec 2.02020
5. 5Whisper2022
6. 6V-JEPA2024
7. 7RT-22023
8. 8Genie2024

All 110 papers, in 19 ladders →