RT-2,
explained.

Robot learning has a data problem the rest of AI doesn't: you can't scrape robot trajectories off the internet. Every demonstration is collected on real hardware, slowly, so robot datasets are tiny next to the web. The result is brittle policies that work only on the exact objects and phrasings they were trained on. Meanwhile, vision-language models trained on the whole internet know an enormous amount about the world — but they can't move a single motor.

RT-2's insight is to fuse the two by making them speak the same language. It takes a large vision-language model and gives it one new thing to say: actions. A robot action is turned into a short string of tokens, drawn from the model's own vocabulary, so producing an action is generation — exactly like producing a caption. Because it's the same model and the same token space, everything the model learned from the web is instantly available when it decides what the robot should do.

A single robot command is a small vector of continuous numbers — a common choice is 7-DoF: three position deltas, three rotation deltas, and one gripper open/close. To make this a string of tokens, RT-2 discretizes each dimension: split its range into a fixed number of bins (say 256), and map the value to the index of the bin it falls in. Each bin index is assigned a token, so a 7-number action becomes 7 tokens — one per dimension.

At inference the model generates these action tokens autoregressively, and they're decoded back to continuous values by mapping each token to its bin's center. The whole action interface is just a fixed vocabulary of bins bolted onto the language model's — no new prediction head, no separate control network. Control has been rewritten as sequence generation.

Because actions are tokens, RT-2 can be trained on a mixture of two very different data sources at once: internet vision-language data (image captioning, visual question answering) and robot trajectories (image + instruction → action tokens). This co-fine-tuning is the crux. The web data keeps the model's vast semantic knowledge alive and sharp; the robot data teaches it to ground that knowledge in motor commands. Crucially they share every weight — there's no separate module where knowledge could get stranded.

The consequence is transfer. When the model sees "pick up the extinct animal" next to a toy dinosaur, it doesn't need a robot demo of that exact task — it already knows from the web what "extinct animal" means and which object matches, and it can route that understanding into action tokens. The robot dataset never contained dinosaurs; the web did. That's why co-training on shared tokens, rather than bolting a controller onto a frozen model, is what makes the semantics flow all the way to the gripper.

The tokenization is exact arithmetic. Split a dimension's range [lo, hi] into B bins; the value x maps to bin index and back to that bin's center:

Δ is one bin width. Every value in a bin decodes to the same center, so decoding is lossy by up to half a bin.

That rounding is the only cost, and it's bounded. The worst-case round-trip error is half a bin width, which shrinks as you add bins — buying finer motor precision at no change to the token count:

Still one token per dimension regardless of B — more bins mean a finer action alphabet, not a longer action string. A few hundred bins is precise enough for manipulation, so the tiny quantization cost buys the enormous benefit of sharing the language model's token space. The runnable version below tokenizes a 7-DoF action and shows the error shrinking with more bins.

RT-2's headline result is emergent generalization — capabilities that appear in the robot only because they came from the web:

These behaviors weren't taught on the robot — they're the web knowledge showing up in motor commands, which is exactly what sharing a token space was supposed to enable. The robot became a way to execute the model's understanding, not a separate thing that had to relearn the world from a few thousand demos.

RT-2 named and popularized the Vision-Language-Action (VLA) model — a single network that maps pixels and language directly to actions — and it made the case that the fastest route to capable robots runs through internet-scale pretraining, not around it. The action-as-token recipe turned robotics into another sequence-modeling problem, so the whole toolkit and scaling curve of large models became available to control. A wave of open VLAs followed the same blueprint.

The deeper lesson echoes Sora's: find the right "token" for your modality and a transformer will learn over it. For video it was the spacetime patch; for control it's the discretized action. Once an action is a token, the boundary between "understanding the world" and "acting in it" gets thin — the same weights that answer a question about an image can command a gripper. RT-2 is the clearest demonstration that robot generalization can be, in large part, inherited rather than collected.