The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Why video is a different beast than images
Image generation is a two-second, single-GPU affair. Video multiplies every hard thing: a 10-second clip is hundreds of frames that must agree with each other — the corgi can't change breed between frames — and a single render holds a top-end GPU for minutes. That combination (temporal consistency × brutal compute) reshapes the entire system around one truth: you are building a render farm with a safety department, not a chat endpoint.
An async job pipeline: screen the prompt while it's cheap, enrich it with an LLM, schedule it onto a GPU fleet with real queue economics, generate in compressed spacetime latents, decode, moderate the actual pixels, stamp provenance, and deliver drafts before finals.
What the new pieces do
- Userclient
- Types "a corgi surfing a wave at golden hour, cinematic" and expects something watchable — eventually, but not silently.
Step 1 · The skeleton
Async or nothing
A render takes 2–10 minutes. An HTTP request that hangs for 10 minutes is a timeout, a retry storm, and a user staring at a spinner wondering if anything is happening. What does the front door look like?
The user submits a prompt. What does the API do?
Minutes-long connections die at every proxy, retry into duplicate renders, and scale catastrophically. Synchronous is simply unavailable at this latency.
The async job pattern: submit → job ID → status stream (queued, position, rendering 40%, done). The user always knows where things stand; the system batches and schedules freely behind it.
Async, yes — but the product needs live progress, previews and cancellation, not a surprise attachment twenty minutes later.
The API/Job Queue accepts the prompt, persists a job, and returns an ID in milliseconds. Status flows back continuously — queue position, progress, preview-ready, done — over polling or a push channel. Cancellation is first-class: users abandon; billing GPU-minutes for unwatched renders is how you lose them twice.
Why this piece earns its place
The hard part of this skeleton is not the queue, it is deciding who owns a job's liveness. A worker has to claim a job and keep claiming it with a heartbeat, because a GPU process that dies at minute seven otherwise leaves a row sitting in rendering forever — no error, no alert, and the first person to notice is the user writing in. Make submission idempotent on a client-supplied key, and be specific about the two details that decide whether that actually works: the key is scoped per user, and the server remembers it for longer than a render takes, or a retry arriving at minute nine starts a second job. Cancellation has to be a flag the worker reads between denoising steps; honoured only at completion it frees no silicon and refunds nothing. This row is also what makes the scheduler's preemption survivable at all — a job can be evicted and requeued precisely because its state lives in a database rather than in a worker's memory. And the failure vocabulary deserves more thought than it gets. Failed is a category, not a state: refused at screening, out of capacity, decoder crash and user-cancelled are four different sentences to the user, they bill differently, and only some of them are worth retrying without asking first.
What the new pieces do
- API / Job Queuebackend
- Accepts the request, returns a job ID immediately, and streams status updates. Nothing about video generation is synchronous.
Step 2 · Screen before you spend
Prompt safety, at the cheap end
Some prompts must never render: abuse material, real-person deepfakes, targeted harassment. And some merely shouldn't: trademark characters, watermark-evasion requests. When do you check — before burning five GPU-minutes, or after?
Where does content policy enforcement start?
Milliseconds of screening before minutes of GPU. Block the clear violations, flag the ambiguous for stricter output review, and refuse identity-deepfake requests outright.
Output moderation is essential (step 6) but catching a violating render AFTER five GPU-minutes is the most expensive possible “no.” Screen twice; spend once.
Generative models don’t refuse; they generate. Policy lives in systems around the model, never in hopes about the model.
Prompt Safety runs before queueing: fast classifiers plus an LLM policy pass for nuance, identity-protection checks (public figures, uploaded faces), and IP filters. Clear violations bounce with a reason; borderline prompts proceed tagged for stricter output moderation downstream.
Why this piece earns its place
Screening sits in front of the job ID, so its latency is the submit latency — and an LLM policy pass is not a millisecond operation. Fire the classifiers and the policy call concurrently rather than in series, and let a confident classifier verdict cancel the LLM call, so the common prompt pays for the cheap check and only the ambiguous middle pays for both. That leaves one genuinely hard decision: what the gate does when the policy model is timing out. A check this cheap next to a render tempts you to fail closed, but refusing everything through a dependency blip turns it into a full outage — and you have a second gate. Fail open, mark the job unscreened, and let output moderation run at its tightest thresholds; you pay GPU-minutes you would rather not, and nothing ships unchecked. Then alarm on that rate, because unscreened is a state you want to see rather than a default you drift into. Refusals, meanwhile, are the one output with no feedback loop — a wrongly refused user does not retry, so refusal rate cannot tell a moved threshold from a real abuse wave. Sample them into human review continuously, not after a complaint.
What the new pieces do
- Prompt Safetyservice
- Screens prompts before any GPU spends: violence, CSAM, real-person deepfakes, IP-protected characters — cheap checks before expensive renders.
Step 3 · Say it like the model heard it
LLM prompt expansion
Users type "corgi surfing." The model was trained on dense, descriptive captions — shot types, lighting, motion, lens language. That gap between how users write and how the model listens is worth more quality than most architecture changes. Who bridges it?
What happens to the user’s terse prompt?
Terse prompts under-specify; the model fills the gaps with mush. "Corgi surfing" deserves camera, light, water physics — details users didn’t know they could ask for.
Rewrite toward the training distribution: shot type, lighting, motion verbs, style. Show users the expansion and let power users opt out — enrichment, not override.
A form with 14 cinematography fields is a product nobody uses. The LLM does the film-school translation invisibly.
The Prompt Expander — an LLM — rewrites the user’s intent into the register the video model was trained on: subject, action, camera, lighting, mood, style. The expansion is shown (and editable), preserving user control while lifting default quality dramatically. This is the same trick DALL·E 3 made famous, and it matters more for video, where motion must be described.
Why this piece earns its place
The expander is the one component that writes text nothing has screened. Safety read what the user typed; the renderer consumes what the expander produced, and a model told to add cinematography will happily supply a named actor, a recognisable logo or a style nobody approved — and a user who wants exactly that can shape the prompt so the expander says it on their behalf. So treat the expansion as an input to policy rather than an output of it, and re-run the cheap classifiers over the expanded caption before it queues. The second thing is that this component has a maintenance schedule nobody puts on a calendar. Its whole job is to sit inside the renderer's training distribution, so every time the renderer is retrained the template drifts out of tune — no error, no alert, just a slow sag in default quality that surfaces months later as vaguer videos and gets blamed on the model. Pin the expander version to the renderer version and re-tune them as a pair. At least that is the easy direction to move: a template edit ships on the next request, while the renderer is a training run.
What the new pieces do
- Prompt Expanderservice
- An LLM rewrites the terse user prompt into the detailed, cinematography-aware description the video model was trained to expect.
Step 4 · The economics layer
Scheduling minutes-long GPU jobs
Every render monopolizes an expensive GPU for minutes. Demand is spiky, supply is fixed-ish, and free users, pro users and enterprise API calls all want the same silicon. This isn’t load balancing — it’s an economy. Who gets the GPUs, when?
How are render jobs assigned to the GPU fleet?
One free user’s 4K batch starves paying customers. FCFS is fair the way a single supermarket line with no express lane is fair — badly.
Video GPUs are scarce, expensive, and slow to provision — capacity moves in weeks, demand moves in minutes. Autoscaling is a supplement, never the answer.
Paid tiers get priority lanes (that’s what they bought), fair-share stops any one user monopolizing a tier, background/batch jobs are preemptible filler, and the queue tells the truth about waits.
The GPU Scheduler runs the economy: priority tiers by plan, fair-share within tiers, preemptible batch soaking idle capacity, admission control that stops accepting jobs it can’t honestly promise, and wait estimates surfaced to the user. Fleet utilization and queue truth are both product features.
Why this piece earns its place
Underneath the tier logic sits a packing problem nobody warns you about. Jobs differ in length and resolution, so a fleet running one undifferentiated queue fragments: a handful of long renders pin whole nodes and a burst of short jobs has nowhere to land even while utilization looks healthy. Duration has to be a scheduling input, which means estimating it before the job runs — from resolution, requested length and model version, fitted on what jobs of that shape actually took. Splitting the fleet into pools by expected duration costs a little utilization and buys back predictability. Preemption has a price the tier table does not show: evicting a render at minute eight throws away eight minutes of silicon, so preempt early in the denoising schedule or checkpoint the latents, and track wasted GPU-minutes as a metric rather than assuming eviction is free. Publish wait estimates from a conservative percentile rather than a mean, too — an estimate that is right on average is wrong for half the people reading it, and they are the half who write in.
What the new pieces do
- GPU Schedulerservice
- Allocates minutes-long GPU jobs: priority tiers, fair-share, preemptible batch, and honest wait estimates. The economics live here.
Back of the envelope
- priority tiers
- enterprise > pro > free — the lanes people pay for
- fair-share
- no single user monopolizes a tier
- preemptible batch
- background jobs fill idle GPU troughs
- honest waits
- shown queue position beats silent hanging
Step 5 · The generator
Diffusion transformers over spacetime latents
Now the actual magic. Generating raw pixels for hundreds of frames is computationally insane, and generating frames independently gives you a flip-book of hallucinations — the surfboard changes color, the wave forgets physics. How do modern systems generate coherent motion at all?
How does the model generate a coherent 10-second clip?
No shared state between frames = no object permanence. The corgi is a different corgi in frame 40. Temporal coherence must be IN the generation, not glued on.
A VAE squeezes video ~100× into latents spanning space AND time; a DiT denoises all patches together, so attention runs across frames — objects persist because generation is joint. Decode returns pixels at the end.
Interpolation yields smooth morphing, not action — nothing new can happen between keyframes. Fine for slideshows; fatal for surfing corgis.
The Video DiT works in a compressed spacetime latent space: a video VAE encodes clips into patches spanning height × width × time, a diffusion transformer denoises all of them jointly — attention flowing across frames — and the Latent Decoder turns finished latents back into pixels. Temporal consistency isn’t a post-process; it’s the architecture.
Why this piece earns its place
Two operational facts about this pair. Memory peaks at the decoder, not the transformer: latents are small and frames are not, so a clip that denoises comfortably can still exhaust VRAM in the decode, which reads to everyone as a random failure late in the job. Decode in frame chunks, and size the machine for the decode rather than the denoise. Second, attention cost grows with the square of the token count, and tokens are frames times resolution — so clip length and resolution are not independent knobs, and the scheduler cannot price a job from either one alone. The VAE is the piece to be careful with. Whatever the decoder cannot represent is baked into every output no matter how good the transformer becomes, and replacing it invalidates a transformer trained against its latent space: a decoder change is a retrain, not a deploy. Which is why the weights version and the seed both belong on the job and beside the stored asset. Renders minutes apart can straddle a rollout and come off different builds, and with neither field recorded you cannot answer the first question anyone asks after a bad deploy — which videos came out of which build — nor re-run a specific render later, since the weights alone will not reproduce it without the seed.
- ~100×compression into latents
- jointdenoising across all frames
- minutesof GPU per clip
What the new pieces do
- Video DiTgpu
- The generator: a diffusion transformer denoising patches of compressed space-time, so objects persist and motion stays coherent across seconds.
- Latent Decoderservice
- Decompresses the finished latents into actual pixels — frames, then an encoded video file.
Step 6 · Check the pixels, sign the work
Output moderation and provenance
The prompt passed screening — but diffusion is stochastic: an innocent prompt can render policy-violating pixels, and a crafted one can smuggle them past text filters. And once released, AI video feeds a misinformation machine unless it's identifiable. Two duties before delivery.
The render finished. What stands between it and the user?
Prompts lie; pixels don’t. Output is the only place policy can actually be verified, and borderline-tagged jobs especially need it.
Vision classifiers sample frames (tighter thresholds for tagged jobs); passing videos get signed provenance (C2PA) and a robust invisible watermark so "is this AI?" stays answerable downstream.
Humans can’t watch a fleet’s output at scale. Automated scans everywhere; humans on the flagged sliver.
Output Moderation scans sampled frames (and audio, if any) with vision classifiers — thresholds tightened for jobs tagged at prompt screening. Clean renders are stamped with C2PA provenance metadata and an invisible watermark that survives re-encoding, so platforms and researchers can identify AI-generated video even after the metadata is stripped.
Why this piece earns its place
Frame sampling is a real decision with a measurable miss rate: a violation can occupy a fraction of a second of a ten-second clip, so a fixed stride steps over it some percentage of the time. Sample denser around cuts and motion spikes, denser again for tagged jobs, and treat the stride as a tuned parameter rather than a constant somebody picked once. Run the same scan on the draft, not only the final — a violating direction caught at draft quality costs seconds of GPU, while the same catch after the full pass costs the whole render and the user waited for a video they were never going to receive. C2PA is a signature, which means you now operate a signing key: rotation, expiry, and a plan for the day it leaks and anything on the internet can claim to be your output. And plan for the re-scan, because policy moves and someone will eventually ask what the archive looks like under the new thresholds. Re-scanning video means decoding all of it again — a render-sized workload competing with live traffic for the same silicon, which is an argument for running it in the preemptible batch lane the scheduler already has, rather than as a script somebody kicks off on a Friday.
What the new pieces do
- Output Moderationservice
- Scans the generated frames (prompts lie; pixels don’t), then embeds C2PA provenance metadata and an invisible watermark.
Step 7 · Deliver drafts, then finals
Progressive rendering and the CDN
Ten minutes is a long time to wait for a video that might be wrong — wrong vibe, wrong motion, wrong corgi. And finished videos need to play instantly, worldwide. The last mile is UX economics: how do users iterate without burning full renders?
How do users iterate toward the video they wanted?
Five iterations × 10 GPU-minutes = an hour of silicon for one usable clip. Exploration at final quality bankrupts the fleet and the user’s patience.
The film-industry pattern: dailies before final cut. A 20-second draft answers "is this the right video?" for 5% of the cost; the winner gets the full render.
Text can’t answer motion questions. Only frames — cheap ones — can.
Draft-then-final: low-resolution, fewer-step preview renders come back in tens of seconds; the user picks a take and only that one gets the full-quality pass. Everything lands in the Asset Store/CDN — previews, finals, thumbnails — served from the edge, with Telemetry (queue depth, GPU-minutes/job, pick-rates) feeding back into scheduler tuning and model evals.
Why this piece earns its place
The storage bill is the part that sneaks up. Drafts outnumber finals by however many takes a user needs, and they are video, so with no retention policy storage grows with usage multiplied by iteration count instead of with usage. Expire drafts on a short clock, keep finals, and make that clock a product decision rather than a default. Delivery has an unusual shape too. A pull-through CDN only starts paying at the second request to a given edge, and a render watched once by the person who made it never gets one: a miss, an origin fetch, an extra hop, then it ages out having served nobody twice. So the number to watch here is origin egress, not hit ratio — the ratio will look dreadful, and that is simply what freshly generated private content does. Save the real caching work — tiered caching, an origin shield, long TTLs — for the assets that get shared, which is a decision you can make at the moment of sharing rather than at write time. Everything stays behind expiring signed URLs regardless, since this content is private by default. And pick-rate is biased in a way worth saying out loud: users pick the best of the drafts they were shown, never the best possible render. It tells you which option won. It cannot tell you the options were good.
- draftsseconds, ~5% of cost
- finalsonly for picked takes
- edge CDNinstant playback worldwide
What the new pieces do
- Asset Store / CDNstore
- Durable storage plus edge delivery: draft previews, final renders, thumbnails — served fast, everywhere.
- Telemetrybus
- Queue depth, GPU-minutes per job, failure rates, and quality/preference signals that tune the scheduler and the models.
The payoff
You built a text-to-video system
From a one-line prompt to a watchable, signed, delivered video: an async job spine, safety at both ends, LLM prompt expansion, a GPU scheduler running real queue economics, diffusion transformers over spacetime latents, provenance watermarking, and draft-then-final rendering through an edge CDN.
Now trigger the viral surge and watch the economy under stress: honest queues, priority lanes, preempted batch, draft-first degradation — the difference between a rough day and a dead product.
Everything you assembled, in order
- Async jobs — job ID in ms, status stream, first-class cancel
- Safety × 2 — cheap prompt screen, truthful pixel scan
- Prompt expansion — LLM translates users into the training distribution
- GPU scheduler — tiers, fair-share, preemptible batch, honest waits
- Spacetime latents — compress ~100×, denoise jointly, decode last
- Provenance — C2PA + invisible watermark on every render
- Draft → final — cheap iteration, expensive only for the winner
- Telemetry — GPU-minutes and pick-rates tune the whole economy