RT-2,
explained.

Rungs 1 to 6 connect modalities to understanding. This rung connects one to acting, and the problem it solves is a mismatch of type. A vision-language model emits text tokens. A robot needs continuous numbers — displacements, rotations, a gripper state. The conventional answer is a separate policy network trained on robot data, which means the control policy learns nothing from the web and knows only the objects in its own small dataset.

The change is almost a trick, and it is the rung. Discretise the action into bins and express those bins as tokens the model's vocabulary already contains. An action becomes a short string. So controlling a robot is text generation, the architecture needs no new output head, and — the part that matters — the model can be trained on robot trajectories and web image-text data together, because both are now the same kind of sequence.

What that buys, stated in units the ladder cares about, is semantic generalisation into control. The model can act on an instruction mentioning an object that appears nowhere in the robot data, because it learned what the word means from the web and the action vocabulary is shared. A policy trained only on robot demonstrations cannot do that at any scale of demonstrations, because the knowledge is simply not in them.

Robot learning has a data problem the rest of AI doesn't: you can't scrape robot trajectories off the internet. Every demonstration is collected on real hardware, slowly, so robot datasets are tiny next to the web. The result is brittle policies that work only on the exact objects and phrasings they were trained on. Meanwhile, vision-language models trained on the whole internet know an enormous amount about the world — but they can't move a single motor.

RT-2's insight is to fuse the two by making them speak the same language. It takes a large vision-language model and gives it one new thing to say: actions. A robot action is turned into a short string of tokens, drawn from the model's own vocabulary, so producing an action is generation — exactly like producing a caption. Because it's the same model and the same token space, everything the model learned from the web is instantly available when it decides what the robot should do.

A single robot command is a small vector of continuous numbers — a common choice is 7-DoF: three position deltas, three rotation deltas, and one gripper open/close. To make this a string of tokens, RT-2 discretizes each dimension: split its range into a fixed number of bins (say 256), and map the value to the index of the bin it falls in. Each bin index is assigned a token, so an action of seven continuous numbers becomes seven tokens — one per dimension — and with the leading end-of-episode flag the model emits eight in all.

At inference the model generates these action tokens autoregressively, and they're decoded back to continuous values by mapping each token to its bin's center. The whole action interface is just a fixed vocabulary of bins bolted onto the language model's — no new prediction head, no separate control network. Control has been rewritten as sequence generation.

Because actions are tokens, RT-2 can be trained on a mixture of two very different data sources at once: internet vision-language data (image captioning, visual question answering) and robot trajectories (image + instruction → action tokens). This co-fine-tuning is the crux. The web data keeps the model's vast semantic knowledge alive and sharp; the robot data teaches it to ground that knowledge in motor commands. Crucially they share every weight — there's no separate module where knowledge could get stranded.

The consequence is transfer. When the model sees "pick up the extinct animal" next to a toy dinosaur, it doesn't need a robot demo of that exact task — it already knows from the web what "extinct animal" means and which object matches, and it can route that understanding into action tokens. The robot dataset never contained dinosaurs; the web did. That's why co-training on shared tokens, rather than bolting a controller onto a frozen model, is what makes the semantics flow all the way to the gripper.

The tokenization is exact arithmetic. Split a dimension's range [lo, hi] into B bins; the value x maps to bin index and back to that bin's center:

Δ is one bin width. Every value in a bin decodes to the same center, so decoding is lossy by up to half a bin.

That rounding is the only cost, and it's bounded. The worst-case round-trip error is half a bin width, which shrinks as you add bins — buying finer motor precision at no change to the token count:

Still one token per dimension regardless of B — more bins mean a finer action alphabet, not a longer action string. A few hundred bins is precise enough for manipulation, so the tiny quantization cost buys the enormous benefit of sharing the language model's token space. The runnable version below tokenizes a 7-DoF action and shows the error shrinking with more bins.

RT-2 adds no new network. It takes two existing vision-language models, writes robot actions as text in their vocabularies, and fine-tunes them on robot demonstrations mixed with their original web data, producing what the paper calls a vision-language-action model.

1. Pick the backbones. PaLI-X, in 5-billion and 55-billion-parameter versions, and PaLM-E at 12 billion. Each encodes the camera image with a Vision Transformer and feeds the resulting tokens, together with the text, to a language model that generates text.
2. Define the action. The robot is commanded in end-effector space: a 6-DoF change in position and rotation, the extension of the gripper, and a discrete command that ends the episode.
3. Discretize. Each continuous dimension is split into 256 uniform bins, so one action becomes 8 integers: the end flag, then the bin indices of the seven continuous values.
4. Map bins to tokens. PaLI-X's tokenizer already has a single token for every integer up to 1,000, so bin k is simply the token for k. PaLM-E has no such tokens, so its 256 least frequently used tokens are reassigned as the action vocabulary.
5. Ask it as a question. Each robot example pairs a camera image with the prompt “Q: what action should the robot take to [instruction]? A:” and expects the 8 integers, separated by spaces, as the answer.
6. Co-fine-tune. The robot data, RT-1's demonstrations collected by 13 robots over 17 months in an office kitchen, is mixed with the backbone's original web vision-language data: robot examples make up about half of the PaLI-X mixture and about two-thirds of the PaLM-E one.
7. Constrain decoding. When the prompt is a robot-action task, sampling is restricted to valid action tokens, so every output decodes to an executable command; on ordinary visual questions the full vocabulary stays open.
8. Run the loop. At each step the current image and the instruction go in, 8 tokens come out, each is mapped back to its bin's center, and the arm moves. The models are served from a multi-TPU cloud service and queried over the network: 1–3 Hz for the 55B model, about 5 Hz for the 5B model.
9. Optionally, plan first. A PaLM-E variant was fine-tuned briefly on data with a short natural-language plan before the action tokens, which let it handle commands that need a step of reasoning, such as choosing a rock to use as an improvised hammer.

What the action string costs

The real cost of writing actions as text is time. The 8 tokens are generated one after another by a model with billions of parameters running in a remote data center, so control runs at a few hertz: RT-1, with 35 million parameters, ran at 3 Hz, and the 55B RT-2 at 1–3 Hz. Later vision-language-action models attacked exactly this. π0, for example, keeps a pretrained VLM but adds a separate flow-matching action expert that outputs continuous chunks of 50 future actions, controlling robots at up to 50 Hz without discretizing anything.

• a — one command; the end flag is already discrete, and the other seven values are binned
• sk — the token for dimension k: an integer token in PaLI-X, a reassigned rare token in PaLM-E
• Lweb — the ordinary next-token loss on the backbone's captioning and question-answering data
• λ — the robot share of the training mixture: about 0.5 for PaLI-X and 0.66 for PaLM-E
• θ — the model's weights, shared by both kinds of data

At the extremes: λ = 1 is plain fine-tuning on robot data, which the paper found generalized worse to new objects and scenes than co-fine-tuning; keeping web data in the mix keeps the model's broader visual knowledge in use while it learns control. λ = 0 learns no control at all. Training the same architecture from scratch on robot data, with no web pretraining, did very poorly. And without the restriction at inference, the model could emit a word in an action slot, which maps to no bin and cannot be executed.

The evaluation ran about 6,000 trials on real robots. The baselines were RT-1 (35 million parameters, trained only on the robot data), VC-1 and R3M (pretrained visual representations feeding an RT-1-style policy), and MOO (an open-vocabulary object detector plus a policy). Tasks were split into those seen in training and unseen ones with new objects, backgrounds or environments, plus emergent tasks that need web knowledge: symbols, reasoning and recognizing people.

Examples of the emergent category include moving an object onto a particular number or icon, or toward a pictured celebrity or a person wearing glasses, none of which appear in the robot data. The ablations isolate where that comes from: the same PaLI-X architecture trained from scratch on robot data performed very poorly, fine-tuning a pretrained model on robot data alone generalized worse than co-fine-tuning with web data, and the 55B model generalized better than the 5B.

The limits are stated plainly. RT-2 learns no new motions: its physical skills are the ones in the robot data, deployed in new semantic situations. And the large models are expensive enough that their control rate may bottleneck tasks that need fast reactions.

What transfers is semantic and not physical, and the paper is clear about the boundary. Web pre-training teaches the model what a banana is and that a wet floor is slippery as a fact; it does not teach the manipulation skill required to pick up something it has never handled. New physical competence still needs robot data, which remains the scarce resource and is the real bottleneck in the field.

Latency is the other hard constraint and it is structural: a large model in a closed control loop must produce each action within the time the controller allows, and model size trades directly against control frequency. Discretisation itself costs precision — a binned action is a quantised action, and fine manipulation feels that.

How it held up: it established vision-language-action models as a category, and actions-as-tokens is the standard trick in it. The broader lesson generalises past robots: if you want a pretrained model's knowledge to reach a new output space, the cheapest route is often to express that space in the model's existing vocabulary rather than to bolt on a new head — the same move as expressing every task as text on the scaling ladder.

RT-2 named and popularized the Vision-Language-Action (VLA) model — a single network that maps pixels and language directly to actions — and it made the case that the fastest route to capable robots runs through internet-scale pretraining, not around it. The action-as-token recipe turned robotics into another sequence-modeling problem, so the whole toolkit and scaling curve of large models became available to control. A wave of open VLAs followed the same blueprint.

The deeper lesson echoes Sora's: find the right "token" for your modality and a transformer will learn over it. For video it was the spacetime patch; for control it's the discretized action. Once an action is a token, the boundary between "understanding the world" and "acting in it" gets thin — the same weights that answer a question about an image can command a gripper. RT-2 is the clearest demonstration that robot generalization can be, in large part, inherited rather than collected.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Joining text to images, audio, video and finally to a robot arm. Starts with the contrastive objective the whole field standardised on.

1. 1CLIP2021
2. 2Flamingo2022
3. 3LLaVA2023
4. 4wav2vec 2.02020
5. 5Whisper2022
6. 6V-JEPA2024
7. 7RT-22023
8. 8Genie2024

All 110 papers, in 19 ladders →