Sora,
explained.

Rung 7 compresses an image into a latent and denoises it. This rung compresses video into a latent volume with time as a third axis, and then makes one further change that is the actual architectural content: instead of denoising that volume with a convolutional network shaped to a fixed resolution, it cuts the volume into spacetime patches and treats them as tokens for a transformer.

Why the patches matter more than the diffusion. A convolutional denoiser is usually trained at one particular shape, so a model trained on square clips of a fixed length tends to generate square clips of that length. A transformer over patches has no fixed input shape at all — a wider video is simply more patches, a longer one is more patches, a taller one is more patches. So one model handles arbitrary resolution, aspect ratio and duration, and the patch count is the compute: cost scales with how much video you asked for, which also means the model can be trained on native, un-cropped material of every shape rather than on everything squashed to one format. This is the patch trick from the vision ladder applied to a volume, and it is the same argument — give up an architectural prior about shape, gain flexibility and the ability to absorb more data.

Stated in the ladder's units: rung 7 made image diffusion cheap enough to be general; this makes video diffusion shape-agnostic, which is what a single model covering the range from a vertical phone clip to a wide minute-long shot requires.

Image and video generators used to be specialists: fixed to a square resolution, a set clip length, trained by cropping every input into the same rigid box. Sora threw that out. It borrows the Vision Transformer's trick — turn an image into a grid of patch tokens — and extends it into time. A video becomes a sequence of spacetime patches, and a transformer learns over that sequence exactly as a language model learns over word tokens.

The consequence is unification. Because a video is "just a list of patches," the model is indifferent to where the patches came from — a tall phone clip, a wide cinematic shot, a two-second loop, or a full minute. Each is simply a different number of tokens. That single representational choice is what lets Sora train on video at its native size and generate at whatever size you ask for. Everything else follows from it.

Raw video is enormous — millions of pixels per second — so Sora works in a latent space, not pixel space. A learned encoder compresses the video both spatially and temporally into a smaller volume of shape (time, height, width). Then that latent volume is cut into a grid of patches; each patch spans a small region of space over a short slice of time, and each becomes one token.

This two-step — compress, then patchify — is the whole encoder story. The compression makes the problem tractable (fewer things to model); the patchify makes it flexible (a variable-length sequence rather than a fixed grid). A generated video is produced by running this in reverse: the transformer denoises a sequence of latent patches, and the decoder turns the finished latent volume back into pixels.

Sora is a diffusion transformer: it combines diffusion (learn to turn noise into data by reversing a gradual noising process) with the transformer architecture (attention over a token sequence) instead of the U-Net that earlier diffusion models used. Training corrupts the spacetime-patch sequence with noise; the transformer is trained to predict and remove that noise, conditioned on a text prompt. At generation time it starts from pure noise patches and denoises step by step into a coherent latent video.

Using a transformer rather than a U-Net matters because transformers scale predictably — the same recipe that made language models better with more compute applies here. Sora's paper leans on exactly this: it shows sample quality improving as training compute grows, illustrated with three qualitative samples at increasing compute. And because a transformer can in principle relate any patch to any other (the report does not disclose the exact attention pattern), the model can plausibly keep consistency across the entire clip — an object that leaves the frame and returns still looks like itself — which is where fixed-window video models tend to fall apart.

Everything hinges on one quantity: how many spacetime patches a video becomes. Start from raw dimensions H × W × frames, apply the encoder's spatial factor s and temporal factor t, then divide by the patch sizes. The sequence length is:

One factor per axis. Duration enters linearly (the first factor); each spatial dimension enters linearly, so resolution — height and width together — enters roughly quadratically.

Now the cost. Self-attention is quadratic in sequence length, so the compute scales as the square of the patch count:

So a video twice as long costs ~4× in attention; one at double resolution costs ~16×. That compounding is why long, high-res video is so expensive — and why the paper treats scaling compute as the main dial for quality. The flip side is the flexibility payoff: the model itself never changes, only the patch count does. The runnable version below computes the latent volume, the patch count, and how it grows.

OpenAI's February 2024 report, Video generation models as world simulators, describes Sora in prose and sample videos. It discloses the shape of the method and almost none of its settings, so each step below says which side of that line it falls on.

1. Compress the video. A video compression network maps raw video to a latent that is smaller in both time and space, and a matching decoder maps generated latents back to pixels. Disclosed: that both exist, and that Sora trains and samples entirely in this latent space. Not disclosed: their architecture, compression factors or training.
2. Cut the latent into spacetime patches. The compressed volume is divided into patches that each span a small region of space over a short stretch of time, and each patch becomes one transformer token. An image is treated as a video with a single frame, so images and videos share one model. Not disclosed: patch sizes or the resulting sequence lengths.
3. Train a diffusion transformer. Given noisy patches plus conditioning such as a text prompt, the model is trained to predict the original clean patches. The report calls it a diffusion transformer and cites the DiT architecture of Peebles and Xie. Not disclosed: parameter count, depth, attention pattern, noise schedule, loss weighting or sampler.
4. Train at native size. Training uses videos and images of varied durations, resolutions and aspect ratios, without resizing, cropping or trimming them to a standard shape as earlier work typically did. The report argues this improves framing, comparing against a version trained on square crops.
5. Re-caption the training data. Following DALL·E 3, OpenAI trained a highly descriptive captioner and used it to caption every training video; the report says training on these captions improved text fidelity and overall quality. At inference, GPT expands a short user prompt into a long, detailed caption of the kind the model was trained on.
6. Generate at any shape. To sample, arrange randomly initialized noise patches in a grid of the desired size and duration, denoise them, and decode. The report shows widescreen 1920×1080 and vertical 1080×1920 video and sizes between, clips up to a minute long, and images up to 2048×2048 from a one-frame grid.
7. Condition on images and video. The same model animates a still image, extends a video forward or backward in time (including into seamless loops), edits a video's style or setting zero-shot with SDEdit, and interpolates between two different videos with a gradual transition.

The scaling evidence

The report's case for the approach is one prompt and seed shown at three training-compute levels, labeled base, 4× and 32×, with quality visibly improving. It is a qualitative demonstration, not a measured scaling curve: the report contains no losses, benchmark scores or compute amounts.

What the demonstrations claim

Under the heading of emerging simulation capabilities, the report shows 3D consistency under camera motion, characters and objects persisting through occlusion, simple interactions that leave lasting effects, such as new paint strokes on a canvas, and Minecraft clips in which Sora controls the player while rendering the world, elicited zero-shot by mentioning Minecraft in the prompt. It attributes these to scale rather than to any explicit 3D or physics machinery.

The report contains no equations. It states that, given noisy patches and conditioning, the model is trained to predict the original clean patches. In the standard form used by diffusion transformers, with clean latent patches x₀, noise ε, a noise level t and the condition c, that target reads:

• x0 — the sequence of clean spacetime patches from the compressed video; its length is the number of patches.
• αt, σt — the noise schedule. Not disclosed.
• w(t) — the per-noise-level loss weight. Not disclosed.
• c — the conditioning: text from the re-captioning pipeline, or image and video frames for animation and extension.
• the identity on the right — predicting clean patches and predicting noise are interchangeable once x_t is known; DiT itself was trained to predict noise.

At the extremes of t, the model either receives nearly clean patches and makes small corrections, or nearly pure noise and must produce the whole clip from the condition. The report gives no step count, does not say whether classifier-free guidance is used, and does not describe how attention is structured across the sequence, so the cost of a given clip cannot be computed from it.

There is no experimental section in the usual sense: no named datasets, no baselines, no metrics and no compute figures. The evidence is a set of sample videos chosen to illustrate each claim, and the one comparison resembling an ablation is the square-crop test. The table separates what is demonstrated from what would be needed to measure it.

The square-crop comparison is the closest thing to an ablation: the model trained on cropped videos sometimes generates subjects only partly in view, while the native-aspect model frames them fully. It is qualitative and shown on selected samples.

The negative results are stated more plainly than the positive ones. The report says the model does not accurately model the physics of many basic interactions, such as glass shattering; that interactions like eating do not always change an object's state correctly; and that long samples can develop incoherence, with objects appearing spontaneously. These failures bear directly on the world-simulator framing of the title: they are cases where the model reproduces what video looks like without what the world does.

Because nothing is measured, the report supports existence claims, that a model built this way can produce these clips, far better than comparative ones, that it beats alternatives by some amount.

The first thing to be clear about is the document. This is a technical report at roughly the level of a detailed blog post — no architecture specifics, no parameter counts, no data description, no training details. The spacetime-patch framing is stated and the rest is withheld, so this rung is read as a design direction with demonstrations, not as a method anyone can reproduce.

The reported failure modes are unusually candid and they are the most informative part. Physical implausibility: objects that appear or vanish, cause and effect that does not follow, a broken glass that reassembles, limbs and objects that pass through each other. Those are not rendering artefacts — they are evidence that the model has learned what video looks like frame to frame without learning what the world does, which bears directly on how much weight to put on the "world simulator" framing attached to models of this kind.

How it held up: diffusion transformers over spacetime patches became the standard video architecture, and the shape-agnostic property is now assumed rather than remarkable. The persuasive part of the release was scale of demonstration rather than an idea nobody had — which is worth naming, because it is the normal shape of a frontier video result and it makes such releases hard to evaluate as contributions.

OpenAI framed Sora not just as a video model but as a step toward a world simulator — the idea that a model which can generate convincing video must, implicitly, be modeling how the world looks and behaves over time. The evidence is suggestive (3D-ish consistency, rough object permanence) rather than conclusive, and the failures are real: physics gets violated, objects appear and vanish, fine causal logic breaks. But the direction matters — it connects video generation to the world-model research program, where the goal is a model you can plan and reason with, not just render from.

The lasting lesson, though, is architectural. Sora is the clearest proof that the transformer-plus-tokens recipe generalizes past text: find the right "token" for your modality — here, the spacetime patch — compress into a latent space, and let a scalable diffusion transformer learn over the sequence. That template now underlies a wave of generative-video systems. The token of video turned out to be a small cube of compressed space and time, and once you have it, video generation becomes another sequence-modeling problem.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Ten papers from the first latent-variable generator to video and real-time 3D. The diffusion half is strictly cumulative — each rung removes a cost the last one paid.

1. 1VAE2014
2. 2GANs2014
3. 3VQ-VAE2017
4. 4DDPM2020
5. 5DDIM2021
6. 6Classifier-Free Guidance2021
7. 7Latent Diffusion2022
8. 8Sora2024
9. 9NeRF2020
10. 103D Gaussian Splatting2023

All 110 papers, in 19 ladders →