The big idea

What makes text-to-image hard?

A user types a sentence and expects a picture. Behind that, a diffusion model runs dozens of iterative denoising steps on a GPU, taking seconds to tens of seconds and a big chunk of very expensive hardware — per image. Now serve that to millions of users whose demand spikes unpredictably, safely, without setting money on fire.

A text-to-image service is really a GPU job system with safety on both ends. The shape: accept a prompt, run it as an async job on a batched, autoscaling GPU fleet, store the result and deliver it via CDN, and moderate both the prompt and the image. The hard parts are latency, cost, and safety — not the web plumbing.

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the pipeline grow, and at the end flood the queue and disable output moderation to see what absorbs the spike and why safety is non-negotiable. Hit Begin.

Step 1 · Why not just return it?

Synchronous generation fails

The obvious API: POST /generate, run the model, return the image in the response. Why does this collapse the moment you have real traffic?

Design decision: A synchronous request that runs the model and returns the image. Why does it fail at scale?

The call: The image is too big to return in one response. — Image size is trivial (and you'd return a URL anyway). The real problem is that generation is slow and GPU-bound, so a synchronous request-per-job model can't hold connections that long or absorb spikes against fixed capacity.

Generation takes seconds to tens of seconds on scarce, expensive GPUs. Holding an HTTP request open for each job ties up resources, hits timeouts, and — fatally — a demand spike instantly overruns your fixed GPU capacity with nowhere to buffer. You must decouple accepting the request from running the slow, capacity-limited work.

Slow + scarce = async: Two properties force an async design: each job is slow (can't hold a connection) and runs on fixed, costly hardware (can't just scale to meet an instantaneous spike). Whenever work is slow and capacity is fixed, you queue it — the same logic as any heavy job system.

Step 2 · Accept fast, work later

The async job API

If you can't return the image inline, what does the API do the instant a prompt arrives?

Make generation an async job. POST /generate validates the request, enqueues a job, and immediately returns a job id. The client then polls GET /jobs/:id or subscribes (WebSocket/SSE/webhook) for completion, receiving the image URL when it's ready. The API is fast and stateless; the slow work happens behind the queue. Track each job's status (queued → running → done/failed) so the client always knows where it stands.

Submit → poll/subscribe → result: The classic long-running-job pattern: accept and acknowledge instantly (202 + job id), do the work asynchronously, and let the client discover the result by polling or a push. This keeps the front door responsive no matter how backed up the GPUs are.

Step 3 · Buffer the burst

A job queue in front of the GPUs

Demand is spiky (a viral moment, a launch) but GPU capacity is fixed and can't scale in seconds. How do you keep a spike from either dropping requests or requiring you to over-provision hugely?

Design decision: Spiky demand, fixed GPU capacity that can't scale instantly. How do you cope?

The call: Put a job queue between the API and the GPU workers; workers pull at their own rate. — The queue absorbs bursts: jobs pile up and drain as GPUs free up, so a spike becomes a longer wait rather than dropped requests or wasted idle capacity. You provision for sustainable throughput, and the queue handles the peaks.

Put a job queue between the API and a pool of GPU workers. The API enqueues; workers pull jobs at their own sustainable rate. A demand spike simply lengthens the queue (and the ETA) instead of dropping requests or forcing you to over-provision GPUs for peak. You size the fleet for steady throughput; the queue absorbs the bursts. This is the shock absorber that makes fixed, costly capacity workable.

Queue decouples demand from capacity: Bursty demand + fixed, expensive supply is exactly what a queue is for. It converts an unservable instantaneous spike into a manageable backlog, lets you run GPUs near full utilization (not idle-for-peak), and gives you a natural place to prioritize and meter jobs.

Step 4 · Inside the box

The diffusion inference loop

A worker has a job — now it actually generates the image. What is the model doing, and why is it so compute-heavy?

The worker runs the diffusion loop: start from random noise in a compressed latent space, then iteratively denoise it over many steps, at each step nudging it toward the text prompt (via cross-attention on the prompt embedding, with a guidance scale controlling how strongly). After the final step, a decoder turns the clean latent into pixels. Each of the dozens of steps is a full neural-network pass — which is why it's slow and GPU-bound. Fewer steps = faster but lower quality; that tradeoff is a key dial.

Iterative denoising: Diffusion generates by reversing a noising process: it learns to remove a little noise at a time, so it takes many sequential model passes to go from noise to a coherent image. Working in a smaller latent space (not full-resolution pixels) is what makes it tractable. Steps × model size = your per-image compute cost.

Step 5 · Feed the fleet

Batched, autoscaling GPU workers

GPUs are most efficient when kept busy with parallel work, they're expensive, and demand rises and falls. How do you run the worker fleet cost-effectively?

Batch multiple generation requests together on each GPU — GPUs are throughput machines, so processing several images in parallel dramatically raises utilization and lowers cost per image. Autoscale the worker pool on queue depth (more workers when the backlog grows, fewer when it drains) to track demand without paying for idle peak capacity. Keep model weights resident (loading a multi-GB model is slow) and manage cold starts by keeping a warm pool. Route different models/resolutions to appropriate GPU types.

Batching + autoscaling: Two levers for GPU economics: batching (fill each GPU with parallel work for max throughput) and autoscaling (match fleet size to the live queue). The enemy is idle expensive hardware and slow cold starts, so you keep GPUs busy and weights warm, and let the queue smooth demand into steady utilization.

Step 6 · Store and deliver

Object store + CDN

The worker produced an image. Where does it go, and how does the user (and anyone who views it later) get it fast and cheaply — without shuttling image bytes back through your job system?

Write the finished image to object storage and record its URL in the job result — the API returns that URL rather than passing image bytes around. Serve the image through a CDN so the user (and any subsequent views, shares, or galleries) fetch it from a nearby edge, cheaply and fast. Generated images are immutable and often re-viewed, making them ideal CDN content. Store thumbnails/variants too if the product needs them.

Result is a URL: Never move large binaries through your queue/API — write them once to durable object storage and pass a URL. A CDN then handles the read fan-out (images get shared and re-viewed), keeping delivery fast globally and cheap, exactly as with any large user-generated media.

Step 7 · Both ends of safety

Prompt and image moderation

Image generation is uniquely risky: a benign-looking prompt can produce harmful, illegal, or NSFW imagery, and users will deliberately try. Filtering text alone is not enough — you have to check what the model actually drew. How do you make it safe?

Design decision: A harmless-looking prompt can still yield harmful imagery. What's required?

The call: Only filter the prompt text. — Prompt-only filtering misses the core risk: the image itself. Innocuous or obfuscated prompts can still yield harmful pixels, so you must moderate the output image, not just the input text.

Moderate on both ends. Input: screen the prompt (policy violations, known evasion tricks) before spending a GPU on it. Output: run every generated image through visual moderation (NSFW / violence / CSAM classifiers) before it's ever delivered — because the model can draw disallowed content from prompts that look harmless. Block, blur, or reject on a hit, and log for review. Add provenance (watermarking / C2PA) so generated images are identifiable. Safety here is mandatory and defense-in-depth.

Two-sided moderation: Text filtering can't catch what a model renders, so image generation needs moderation at the image layer too. Input moderation saves compute and blocks intent; output moderation is the last line before a user sees harmful pixels. For the most serious categories, this is a legal and platform necessity, not a nice-to-have.

Step 8 · The sharp edges

Cost, priority & cold starts

The realities of running expensive generative infra: GPU cost dominates everything, free and paid users compete for the same fleet, latency varies, and identical prompts get regenerated.

Control cost aggressively — GPU time is the bill — with batching, right-sized GPUs, step-count tuning, and per-user quotas. Prioritize the queue (paid/interactive jobs jump ahead of free/batch) so premium users get low latency while free traffic fills spare capacity. Manage cold starts with warm pools and by keeping popular models resident. Cache/dedupe identical prompt+seed requests. Set honest ETAs and stream progress (or intermediate previews) so a multi-second wait feels responsive. And handle failures with idempotent retries that don't double-charge GPU time.

Design for the unhappy path: GPU cost → batch, tune steps, quota. Mixed tiers → priority queue. Latency swings → warm pools, previews, honest ETAs. Repeat prompts → cache. Every lever here trades against the one resource that dominates a generative service: expensive GPU seconds. Optimizing them is the business.

You did it

You just designed a text-to-image service.

• S — y
• M — a
• A —
• T — h
• B — a
• W — r
• M — o