Genie,
explained.

Genie is a generative interactive environment. Prompt it with a single image — a sketch, a photo, a game screenshot — and it produces a playable 2D world: at each step you provide an action, and it generates the next frame conditioned on your choice. Unlike a video generator that plays a fixed clip, Genie is interactive — the future depends on what you do, frame by frame.

What makes that remarkable is the training data. Genie learned from a huge corpus of unlabeled internet gameplay and video, where the actions the original players took were never recorded. There's no "player pressed right" annotation anywhere. Yet the trained model exposes a small, consistent set of controls. Genie had to discover what the actions even were — and that discovery, done unsupervised, is the paper's central contribution.

Normally, to train a controllable model you need paired data: state, action, next state. Robotics collects it painfully on hardware; RT-2 needed robot trajectories with the commands recorded. But internet video is the opposite — endless frames, zero action labels. If you could learn control from that, you'd unlock the largest dataset in existence for training interactive models and agents.

The obstacle is obvious: without labels, how does the model know what "an action" is? Genie's answer is to treat the action as a hidden variable to be inferred. Between two consecutive frames, something changed the world — call that something the latent action, and learn to recover it from the frames themselves. The model is never told the action; it's asked to explain the transition, and the explanation becomes the action.

Genie has three pieces. A video tokenizer compresses frames into tokens (like Sora's patches). A latent action model (LAM) looks at consecutive frames and infers a discrete latent action explaining the transition. A dynamics model predicts the next frame from the current frame plus a latent action. Trained together on video, they let you swap in your chosen latent action at play time.

The magic is the discrete bottleneck. If the latent action could be any high-dimensional vector, it would just smuggle the whole next frame across and learn nothing reusable. Forcing it through a tiny codebook — only a handful of possible codes — makes the model compress each transition into one of a few reusable actions. To reconstruct many transitions with so few codes, each code has to capture a consistent kind of change (a "move right", a "jump"). That consistency is what turns inferred latents into genuine controls: at inference, the LAM is dropped and you supply the latent action yourself.

Strip it to essentials. Let the LAM infer the latent action as the transition between frames, and the dynamics model apply an action to a state. Training over a video discovers the set of latent actions actually used — the codebook:

The bottleneck keeps the codebook tiny, so a few reusable latent actions must explain all transitions — which is exactly what makes them controls.

Two properties make the latents usable. They round-trip: infer the action from a transition, apply it, and you recover that transition. And they're consistent: the same code produces the same kind of change regardless of state — the property that lets a player treat a code as a button:

Round-trip means the action space is a faithful code for transitions; consistency means each code is a reusable control. The runnable version below discovers a small codebook from unlabeled transitions and plays a world forward with chosen actions.

Genie demonstrated that action-controllable world models can be learned from action-free video at scale:

That the latent actions transfer across different prompted worlds is the striking part: an action learned from one kind of video behaves sensibly in a scene the model built from a fresh image. The discrete bottleneck didn't just memorize transitions — it distilled a small, general vocabulary of "ways a world can change."

Genie's importance is the door it opens. If control can be learned from unlabeled video, then the path to foundation world models — and to training agents inside generated environments — no longer depends on scarce action-labeled data. You could, in principle, generate endless interactive worlds for an agent to practice in, each conjured from an image. It reframes the abundant, label-free video of the internet as training data for interaction, not just perception.

It also threads two research lines together. From generative video it takes the ability to synthesize frames; from model-based RL it takes the idea of a learned world you can act inside — but it earns its action space unsupervised rather than being handed one. Whether latent-action world models become the substrate for training general agents is still an open bet, but Genie made the most audacious part — inventing the controls from label-free video — concrete and reproducible.