Sora,
explained.

Image and video generators used to be specialists: fixed to a square resolution, a set clip length, trained by cropping every input into the same rigid box. Sora threw that out. It borrows the Vision Transformer's trick — turn an image into a grid of patch tokens — and extends it into time. A video becomes a sequence of spacetime patches, and a transformer learns over that sequence exactly as a language model learns over word tokens.

The consequence is unification. Because a video is "just a list of patches," the model is indifferent to where the patches came from — a tall phone clip, a wide cinematic shot, a two-second loop, or a full minute. Each is simply a different number of tokens. That single representational choice is what lets Sora train on video at its native size and generate at whatever size you ask for. Everything else follows from it.

Raw video is enormous — millions of pixels per second — so Sora works in a latent space, not pixel space. A learned encoder compresses the video both spatially and temporally into a smaller volume of shape (time, height, width). Then that latent volume is cut into a grid of patches; each patch spans a small region of space over a short slice of time, and each becomes one token.

This two-step — compress, then patchify — is the whole encoder story. The compression makes the problem tractable (fewer things to model); the patchify makes it flexible (a variable-length sequence rather than a fixed grid). A generated video is produced by running this in reverse: the transformer denoises a sequence of latent patches, and the decoder turns the finished latent volume back into pixels.

Sora is a diffusion transformer: it combines diffusion (learn to turn noise into data by reversing a gradual noising process) with the transformer architecture (attention over a token sequence) instead of the U-Net that earlier diffusion models used. Training corrupts the spacetime-patch sequence with noise; the transformer is trained to predict and remove that noise, conditioned on a text prompt. At generation time it starts from pure noise patches and denoises step by step into a coherent latent video.

Using a transformer rather than a U-Net matters because transformers scale predictably — the same recipe that made language models better with more compute applies here. Sora's paper leans on exactly this: it shows sample quality and temporal coherence improving smoothly as training compute grows. And because attention is global over the whole patch sequence, the model can enforce consistency across the entire clip — an object that leaves the frame and returns still looks like itself — which is where fixed-window video models tend to fall apart.

Everything hinges on one quantity: how many spacetime patches a video becomes. Start from raw dimensions H × W × frames, apply the encoder's spatial factor s and temporal factor t, then divide by the patch sizes. The sequence length is:

One factor per axis. Duration enters linearly (the first factor); each spatial dimension enters linearly, so resolution — height and width together — enters roughly quadratically.

Now the cost. Self-attention is quadratic in sequence length, so the compute scales as the square of the patch count:

So a video twice as long costs ~4× in attention; one at double resolution costs ~16×. That compounding is why long, high-res video is so expensive — and why the paper treats scaling compute as the main dial for quality. The flip side is the flexibility payoff: the model itself never changes, only the patch count does. The runnable version below computes the latent volume, the patch count, and how it grows.

Sora's report is a technical demonstration rather than a benchmark paper, but its claims reset expectations for generative video:

Those emergent behaviors — a coffee cup that stays put, a painter whose strokes persist on the canvas — arise from scale and the global-attention representation, not from any hand-coded physics. They're imperfect (objects still morph and flicker), but they're the reason the paper reaches for a bigger claim than "video generator."

OpenAI framed Sora not just as a video model but as a step toward a world simulator — the idea that a model which can generate convincing video must, implicitly, be modeling how the world looks and behaves over time. The evidence is suggestive (3D-ish consistency, rough object permanence) rather than conclusive, and the failures are real: physics gets violated, objects appear and vanish, fine causal logic breaks. But the direction matters — it connects video generation to the world-model research program, where the goal is a model you can plan and reason with, not just render from.

The lasting lesson, though, is architectural. Sora is the clearest proof that the transformer-plus-tokens recipe generalizes past text: find the right "token" for your modality — here, the spacetime patch — compress into a latent space, and let a scalable diffusion transformer learn over the sequence. That template now underlies a wave of generative-video systems. The token of video turned out to be a small cube of compressed space and time, and once you have it, video generation becomes another sequence-modeling problem.