The big idea

Why video is a different beast than images

Image generation is a two-second, single-GPU affair. Video multiplies every hard thing: a 10-second clip is hundreds of frames that must agree with each other — the corgi can't change breed between frames — and a single render holds a top-end GPU for minutes. That combination (temporal consistency × brutal compute) reshapes the entire system around one truth: you are building a render farm with a safety department, not a chat endpoint.

An async job pipeline: screen the prompt while it's cheap, enrich it with an LLM, schedule it onto a GPU fleet with real queue economics, generate in compressed spacetime latents, decode, moderate the actual pixels, stamp provenance, and deliver drafts before finals.

How to read this: Each step opens with a real design decision — make the call before seeing what ships. Watch the farm assemble, hover any box, replay the flow. At the end, trigger a viral surge and watch queue physics take over. Hit Begin.

Step 1 · The skeleton

Async or nothing

A render takes 2–10 minutes. An HTTP request that hangs for 10 minutes is a timeout, a retry storm, and a user staring at a spinner wondering if anything is happening. What does the front door look like?

Design decision: The user submits a prompt. What does the API do?

The call: Return a job ID instantly; the client polls or subscribes for status and progress. — The async job pattern: submit → job ID → status stream (queued, position, rendering 40%, done). The user always knows where things stand; the system batches and schedules freely behind it.

The API/Job Queue accepts the prompt, persists a job, and returns an ID in milliseconds. Status flows back continuously — queue position, progress, preview-ready, done — over polling or a push channel. Cancellation is first-class: users abandon; billing GPU-minutes for unwatched renders is how you lose them twice.

Jobs, not requests: The moment work outlives an HTTP timeout, you're a job system: persistent state, idempotent submission, progress events, cancellation. Every long-running AI product converges on this skeleton.

Step 2 · Screen before you spend

Prompt safety, at the cheap end

Some prompts must never render: abuse material, real-person deepfakes, targeted harassment. And some merely shouldn't: trademark characters, watermark-evasion requests. When do you check — before burning five GPU-minutes, or after?

Design decision: Where does content policy enforcement start?

The call: At the prompt — cheap classifiers and an LLM policy check before the job ever queues. — Milliseconds of screening before minutes of GPU. Block the clear violations, flag the ambiguous for stricter output review, and refuse identity-deepfake requests outright.

Prompt Safety runs before queueing: fast classifiers plus an LLM policy pass for nuance, identity-protection checks (public figures, uploaded faces), and IP filters. Clear violations bounce with a reason; borderline prompts proceed tagged for stricter output moderation downstream.

Two gates, one policy: Prompt screening is cheap but foolable (prompts lie); output scanning is truthful but expensive (pixels don’t lie). Production systems run both, sharing one policy — the tag from gate one tightens the threshold at gate two.

Step 3 · Say it like the model heard it

LLM prompt expansion

Users type "corgi surfing." The model was trained on dense, descriptive captions — shot types, lighting, motion, lens language. That gap between how users write and how the model listens is worth more quality than most architecture changes. Who bridges it?

Design decision: What happens to the user’s terse prompt?

The call: An LLM expands it into the detailed, cinematography-aware caption the model expects — visibly, and reversibly. — Rewrite toward the training distribution: shot type, lighting, motion verbs, style. Show users the expansion and let power users opt out — enrichment, not override.

The Prompt Expander — an LLM — rewrites the user’s intent into the register the video model was trained on: subject, action, camera, lighting, mood, style. The expansion is shown (and editable), preserving user control while lifting default quality dramatically. This is the same trick DALL·E 3 made famous, and it matters more for video, where motion must be described.

Meet the training distribution: A generative model performs best on inputs shaped like its training data. Prompt expansion is distribution-matching as a product feature — the cheapest quality win in the whole pipeline.

Step 4 · The economics layer

Scheduling minutes-long GPU jobs

Every render monopolizes an expensive GPU for minutes. Demand is spiky, supply is fixed-ish, and free users, pro users and enterprise API calls all want the same silicon. This isn’t load balancing — it’s an economy. Who gets the GPUs, when?

Design decision: How are render jobs assigned to the GPU fleet?

The call: Priority tiers with fair-share inside each, preemptible batch filling the idle troughs, and honest wait estimates. — Paid tiers get priority lanes (that’s what they bought), fair-share stops any one user monopolizing a tier, background/batch jobs are preemptible filler, and the queue tells the truth about waits.

The GPU Scheduler runs the economy: priority tiers by plan, fair-share within tiers, preemptible batch soaking idle capacity, admission control that stops accepting jobs it can’t honestly promise, and wait estimates surfaced to the user. Fleet utilization and queue truth are both product features.

Scarcity is the design input: When each job costs dollars of GPU time, the scheduler IS the business model: tiers price the queue, previews (next step) cheapen exploration, and utilization decides gross margin. Design it like an exchange, not a load balancer.

Step 5 · The generator

Diffusion transformers over spacetime latents

Now the actual magic. Generating raw pixels for hundreds of frames is computationally insane, and generating frames independently gives you a flip-book of hallucinations — the surfboard changes color, the wave forgets physics. How do modern systems generate coherent motion at all?

Design decision: How does the model generate a coherent 10-second clip?

The call: Compress video into spacetime latent patches, denoise them jointly with a diffusion transformer, then decode. — A VAE squeezes video ~100× into latents spanning space AND time; a DiT denoises all patches together, so attention runs across frames — objects persist because generation is joint. Decode returns pixels at the end.

The Video DiT works in a compressed spacetime latent space: a video VAE encodes clips into patches spanning height × width × time, a diffusion transformer denoises all of them jointly — attention flowing across frames — and the Latent Decoder turns finished latents back into pixels. Temporal consistency isn’t a post-process; it’s the architecture.

Compress, then generate: Two decisions do all the work: operate in latents (≈100× cheaper than pixels) and denoise space-time jointly (coherence by construction). Every frontier video model — Sora-class onward — stands on this pair.

Step 6 · Check the pixels, sign the work

Output moderation and provenance

The prompt passed screening — but diffusion is stochastic: an innocent prompt can render policy-violating pixels, and a crafted one can smuggle them past text filters. And once released, AI video feeds a misinformation machine unless it's identifiable. Two duties before delivery.

Design decision: The render finished. What stands between it and the user?

The call: Frame-level moderation scans, then C2PA provenance metadata + an invisible watermark embedded in the file. — Vision classifiers sample frames (tighter thresholds for tagged jobs); passing videos get signed provenance (C2PA) and a robust invisible watermark so "is this AI?" stays answerable downstream.

Output Moderation scans sampled frames (and audio, if any) with vision classifiers — thresholds tightened for jobs tagged at prompt screening. Clean renders are stamped with C2PA provenance metadata and an invisible watermark that survives re-encoding, so platforms and researchers can identify AI-generated video even after the metadata is stripped.

Provenance is a public good: Watermarking doesn’t stop a determined adversary — it keeps the honest ecosystem honest: platforms can label, journalists can verify, and your product isn’t the anonymous engine of the next viral fake.

Step 7 · Deliver drafts, then finals

Progressive rendering and the CDN

Ten minutes is a long time to wait for a video that might be wrong — wrong vibe, wrong motion, wrong corgi. And finished videos need to play instantly, worldwide. The last mile is UX economics: how do users iterate without burning full renders?

Design decision: How do users iterate toward the video they wanted?

The call: Cheap fast drafts first (low-res, fewer steps); full quality only on the take the user picks. — The film-industry pattern: dailies before final cut. A 20-second draft answers "is this the right video?" for 5% of the cost; the winner gets the full render.

Draft-then-final: low-resolution, fewer-step preview renders come back in tens of seconds; the user picks a take and only that one gets the full-quality pass. Everything lands in the Asset Store/CDN — previews, finals, thumbnails — served from the edge, with Telemetry (queue depth, GPU-minutes/job, pick-rates) feeding back into scheduler tuning and model evals.

Previews are the profit margin: Iteration is where users live and where GPUs die. Making exploration 20× cheaper than finals is simultaneously the best UX decision and the best unit-economics decision in the system — one mechanism, both wins.

The payoff

You built a text-to-video system

From a one-line prompt to a watchable, signed, delivered video: an async job spine, safety at both ends, LLM prompt expansion, a GPU scheduler running real queue economics, diffusion transformers over spacetime latents, provenance watermarking, and draft-then-final rendering through an edge CDN.

Now trigger the viral surge and watch the economy under stress: honest queues, priority lanes, preempted batch, draft-first degradation — the difference between a rough day and a dead product.

• Async jobs — job ID in ms, status stream, first-class cancel
• Safety × 2 — cheap prompt screen, truthful pixel scan
• Prompt expansion — LLM translates users into the training distribution
• GPU scheduler — tiers, fair-share, preemptible batch, honest waits
• Spacetime latents — compress ~100×, denoise jointly, decode last
• Provenance — C2PA + invisible watermark on every render
• Draft → final — cheap iteration, expensive only for the winner
• Telemetry — GPU-minutes and pick-rates tune the whole economy