Genie,
explained.

Rung 7 needs trajectories in which someone recorded what action was taken, and that is the scarcest data in embodied learning — every hour of it requires a robot, an operator and a rig. Meanwhile the internet holds an effectively unlimited quantity of video of things happening, with no action labels whatsoever.

The change is to infer the actions. A latent action model looks at consecutive frames and learns a small discrete set of latent actions that explains the transition between them — whatever changed between this frame and the next is attributed to one of a handful of learned codes, and those codes are shaped by being required to make the next frame predictable. Nobody says what the actions are; the constraint that there are few of them and that they must carry the transition is what makes them meaningful.

A dynamics model then takes a frame and a latent action and produces the next frame, which turns the whole thing into something playable: a user supplies one of the discrete action codes, the world responds, frame by frame. Stated in the ladder's units, that is a controllable environment learned entirely from passive, unlabelled video — the action labels rung 7 depends on are not collected, they are discovered.

Genie is a generative interactive environment. Prompt it with a single image — a sketch, a photo, a game screenshot — and it produces a playable 2D world: at each step you provide an action, and it generates the next frame conditioned on your choice. Unlike a video generator that plays a fixed clip, Genie is interactive — the future depends on what you do, frame by frame.

What makes that remarkable is the training data. Genie learned from a huge corpus of unlabeled internet gameplay and video, where the actions the original players took were never recorded. There's no "player pressed right" annotation anywhere. Yet the trained model exposes a small, consistent set of controls. Genie had to discover what the actions even were — and that discovery, done unsupervised, is the paper's central contribution.

Normally, to train a controllable model you need paired data: state, action, next state. Robotics collects it painfully on hardware; RT-2 needed robot trajectories with the commands recorded. But internet video is the opposite — endless frames, zero action labels. If you could learn control from that, you'd unlock the largest dataset in existence for training interactive models and agents.

The obstacle is obvious: without labels, how does the model know what "an action" is? Genie's answer is to treat the action as a hidden variable to be inferred. Between two consecutive frames, something changed the world — call that something the latent action, and learn to recover it from the frames themselves. The model is never told the action; it's asked to explain the transition, and the explanation becomes the action.

Genie has three pieces. A video tokenizer compresses frames into tokens (like Sora's patches). A latent action model (LAM) looks at consecutive frames and infers a discrete latent action explaining the transition. A dynamics model predicts the next frame from the current frame plus a latent action. Trained together on video, they let you swap in your chosen latent action at play time.

The magic is the discrete bottleneck. If the latent action could be any high-dimensional vector, it would just smuggle the whole next frame across and learn nothing reusable. Forcing it through a tiny codebook — only a handful of possible codes — makes the model compress each transition into one of a few reusable actions. To reconstruct many transitions with so few codes, each code has to capture a consistent kind of change (a "move right", a "jump"). That consistency is what turns inferred latents into genuine controls: at inference, the LAM is dropped and you supply the latent action yourself.

Strip it to essentials. Let the LAM infer the latent action as the transition between frames, and the dynamics model apply an action to a state. Training over a video discovers the set of latent actions actually used — the codebook:

The bottleneck keeps the codebook tiny, so a few reusable latent actions must explain all transitions — which is exactly what makes them controls.

Two properties make the latents usable. They round-trip: infer the action from a transition, apply it, and you recover that transition. And they're consistent: the same code produces the same kind of change regardless of state — the property that lets a player treat a code as a button:

Round-trip means the action space is a faithful code for transitions; consistency means each code is a reusable control. The runnable version below discovers a small codebook from unlabeled transitions and plays a world forward with chosen actions.

Genie is three networks trained on 30,000 hours of 2D platformer gameplay with no actions, rewards or text. One compresses video into tokens, one invents actions, and one predicts the next frame from an action. Only the first and last are needed to play.

1. Build the dataset. Public videos matching platformer keywords came to about 244,000 hours. An 11M-parameter classifier, trained on about 10,000 hand-labeled videos, filtered them down to 6.8 million 16-second clips, 30,000 hours, at 10 frames per second and 160 × 90 pixels.
2. Use one block type throughout. All three networks are built from spatiotemporal (ST) Transformer blocks: attention within each frame, then causal attention across frames at each position, then a single feed-forward layer. Their cost grows linearly with the number of frames rather than quadratically.
3. Tokenize the video. ST-ViViT, a 200M-parameter VQ-VAE with ST blocks in its encoder and decoder, maps each frame to a grid of discrete tokens from a 1,024-entry codebook, with patch size 4. It is causal in time, so a frame's tokens depend only on that frame and earlier ones. It is trained first.
4. Learn latent actions. The 300M-parameter latent action model (LAM) has an encoder that reads all previous frames and the next frame and outputs one latent per transition, quantized to one of only 8 codes, and a decoder that must reconstruct the next frame from the previous frames and that code alone. Because the decoder never sees the next frame, information about it can arrive only through a 3-bit code, which forces the code to describe the change between frames. The LAM works on raw pixels.
5. Learn the dynamics. A 10.1B-parameter decoder-only MaskGIT Transformer takes the tokens of past frames plus the latent actions, added to the token embeddings with gradients stopped, and predicts the next frame's tokens with a cross-entropy loss. During training a random 50 to 100% of its input tokens is masked.
6. Train the pair together. After the tokenizer, the LAM (on pixels) and the dynamics model (on tokens) are co-trained. The final dynamics model trained for 125,000 steps at batch size 512 on 256 TPUv5p chips.
7. Play. Prompt with any single image: a game frame, a sketch or a photo. A player picks one of the 8 codes; the dynamics model generates the next frame's tokens in 25 MaskGIT steps; the tokenizer decodes them; and the frame joins the history. Apart from its codebook, the LAM is thrown away.

The pieces total about 10.7 billion parameters, which the paper rounds to 11B. Each new frame is sampled with 25 MaskGIT steps at temperature 2, so generation runs at about one frame per second, and the model remembers only the last 16 frames. Two small choices carry a lot of weight. The latent action enters the dynamics model as an added embedding rather than as extra tokens, which the paper reports made the generated worlds more controllable. And its gradient is stopped, so the dynamics loss cannot reshape what the codes mean; only the latent action model's own reconstruction does that.

• x1:t — the frames so far; xt+1 — the next frame, seen only by the LAM's encoder
• ut — the encoder's continuous output; ek — the nearest of the 8 code vectors, which becomes ã_t
• sg — stop-gradient; β — the commitment weight of the standard VQ-VAE objective that the paper adopts
• z — the tokenizer's discrete frame tokens; the dynamics loss covers the masked positions
• x̂t — a frame generated with latent actions inferred from the real clip; x̂′t — one generated with random latent actions

At the extremes: with a single code the latent carries no information, the dynamics model learns to ignore it, and Δ_tPSNR is about zero, so there is no control. With a huge or continuous code, the latent could carry the next frame itself; reconstruction would be easy, and the codes would stop meaning actions. Eight codes sit between: too few to smuggle a frame across, enough for a handful of moves. The paper reports Δ_tPSNR at t = 4.

The paper measures two things: video quality with FVD (lower is better) and controllability with Δ_tPSNR (higher is better), on the platformer data and on a robotics set built largely from RT-1 videos with the actions removed. Dynamics models were scaled from 40 million to 2.7 billion parameters, and training loss fell steadily with size and with batch size, before the final 10.1B run.

Reading pixels made the actions far more controllable at a small cost in platformer video quality; the paper's explanation is that tokenizing can discard some of the motion information an action has to explain. On robot video, pixels helped on both measures, and a 2.5B model trained on those videos learned consistent latent actions such as up, down and left without ever seeing a command.

The strongest result is for agents. In CoinRun, a policy trained to imitate expert videos labeled only with Genie's inferred latent actions matched an oracle behavioral-cloning agent trained with the real actions, once as few as 200 expert samples were used to map the codes onto real controls. The negatives are on the record too: generation runs at about one frame per second, memory spans 16 frames, and the model can hallucinate unrealistic futures.

The latent actions are unnamed and not guaranteed to be consistent: nothing ties code three to "jump" across contexts, nothing makes the set complete, and nothing makes them interpretable — so the control interface is discovered rather than designed, and a user has to find out what each code does. Generation is frame by frame, so it is slow and drifts: errors accumulate over a long rollout and the world degrades.

The most important thing to be clear about is what kind of object this is. It is a generative model of video, not a simulator — it produces frames that look like plausible consequences, with no physical state and no guarantee of consistency, so an object can fail to persist and a consequence can fail to follow. That is the same caution MuZero earns from the opposite direction: a learned model can be useful without being faithful, and if you plan against it, search will find precisely the places where it is wrong.

What would settle it: whether latent actions inferred from video are good enough to train a policy that then works in the real world, which is the only claim that would matter and is not established by the world being playable. The idea is the contribution and it is a strong one — the scarcest label in embodied learning may be inferable from the abundant unlabelled data — and the ladder ends here rather than on something more finished because that is honestly where this line is.

Genie's importance is the door it opens. If control can be learned from unlabeled video, then the path to foundation world models — and to training agents inside generated environments — no longer depends on scarce action-labeled data. You could, in principle, generate endless interactive worlds for an agent to practice in, each conjured from an image. It reframes the abundant, label-free video of the internet as training data for interaction, not just perception.

It also threads two research lines together. From generative video it takes the ability to synthesize frames; from model-based RL it takes the idea of a learned world you can act inside — but it earns its action space unsupervised rather than being handed one. Whether latent-action world models become the substrate for training general agents is still an open bet, but Genie made the most audacious part — inventing the controls from label-free video — concrete and reproducible.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Joining text to images, audio, video and finally to a robot arm. Starts with the contrastive objective the whole field standardised on.

1. 1CLIP2021
2. 2Flamingo2022
3. 3LLaVA2023
4. 4wav2vec 2.02020
5. 5Whisper2022
6. 6V-JEPA2024
7. 7RT-22023
8. 8Genie2024

All 110 papers, in 19 ladders →