The whole design, in writing
Learn AI system design by building a text-to-image generation service like Midjourney, DALL·E or a hosted Stable Diffusion step by step. An interactive guide covering why synchronous generation fails, the async job pattern, the diffusion inference loop, a batched autoscaling GPU worker fleet, storing and delivering images via object store + CDN, input and output safety moderation, and the unhappy paths (cost, queue priority, cold starts).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What makes text-to-image hard?
A user types a sentence and expects a picture. Behind that, a diffusion model runs dozens of iterative denoising steps on a GPU, taking seconds to tens of seconds and a big chunk of very expensive hardware — per image. Now serve that to millions of users whose demand spikes unpredictably, safely, without setting money on fire.
A text-to-image service is really a GPU job system with safety on both ends. The shape: accept a prompt, run it as an async job on a batched, autoscaling GPU fleet, store the result and deliver it via CDN, and moderate both the prompt and the image. The hard parts are latency, cost, and safety — not the web plumbing.
What the new pieces do
- Userclient
- Someone typing "an astronaut riding a horse, oil painting". Wants an image back — and doesn't mind waiting a few seconds.
Step 1 · Why not just return it?
Synchronous generation fails
The obvious API: POST /generate, run the model, return the image in the response. Why does this collapse the moment you have real traffic?
A synchronous request that runs the model and returns the image. Why does it fail at scale?
Each generation occupies an expensive GPU for seconds to tens of seconds; holding the HTTP request open the whole time ties up server resources, times out, and means a traffic burst instantly exceeds fixed GPU capacity with no way to buffer. You need to decouple accepting the request from doing the work.
Image size is trivial (and you'd return a URL anyway). The real problem is that generation is slow and GPU-bound, so a synchronous request-per-job model can't hold connections that long or absorb spikes against fixed capacity.
HTTP carries anything. The issue is duration and capacity: a multi-second GPU job per request doesn't fit the synchronous request/response model under load.
Generation takes seconds to tens of seconds on scarce, expensive GPUs. Holding an HTTP request open for each job ties up resources, hits timeouts, and — fatally — a demand spike instantly overruns your fixed GPU capacity with nowhere to buffer. You must decouple accepting the request from running the slow, capacity-limited work.
Why this piece earns its place
The first thing to break under a synchronous design isn't the GPU — it's whatever sits in front of it. Load balancers and proxies carry idle timeouts shorter than a slow generation, and when one fires the client's connection dies while the worker keeps denoising: you pay for that image in full and nobody receives it. Worse, a client that times out retries, so at the exact moment you are over capacity every user doubles their own load on the scarcest thing you own — congestive collapse on hardware you cannot add in seconds. Skip the pattern when the numbers genuinely allow it, on an internal tool with a handful of users, where a queue is machinery you maintain for nothing. The deeper reason to draw the boundary anyway is that synchronous work is anonymous. Once a generation is a job with an id it becomes something the rest of the system can name: cancelled when the user closes the tab, attributed to whoever pays for it, pointed at by the status record, by the stored image and by the audit trail. A request in flight gives you nothing to attach any of that to — when the socket dies, every trace of what the user wanted dies with it.
Step 2 · Accept fast, work later
The async job API
If you can't return the image inline, what does the API do the instant a prompt arrives?
Make generation an async job. POST /generate validates the request, enqueues a job, and immediately returns a job id. The client then polls GET /jobs/:id or subscribes (WebSocket/SSE/webhook) for completion, receiving the image URL when it's ready. The API is fast and stateless; the slow work happens behind the queue. Track each job's status (queued → running → done/failed) so the client always knows where it stands.
Why this piece earns its place
The load nobody sizes for is the status path. One generation is a single write, but the client asks after it again and again until the picture lands, so reads dominate writes by a wide margin and each one has to be a single lookup by primary key — never a scan, never a join across the queue. Hand clients an explicit poll interval and widen it as the wait grows, or push. If you push, the record still has to be authoritative and the socket only an accelerator: a client that reconnects must be able to ask where things stand and get the same answer, or a dropped connection becomes a job that finished for nobody. Then decide what the record says when things go wrong. failed is not a state, it is a family — a rejected prompt, an out-of-memory worker and a malformed request want different reactions, so the terminal state carries a reason and whether retrying is worth anything. And because that record holds the prompt and later the image URL, the job id is a capability: make it unguessable and check ownership on every status read, not just on submit.
- 202 + job idinstant accept
- poll / subscribeget the result
- statusqueued → done
What the new pieces do
- APIgateway
- Accepts a generation request, returns a job id immediately, and lets the client poll or subscribe for the result.
Step 3 · Buffer the burst
A job queue in front of the GPUs
Demand is spiky (a viral moment, a launch) but GPU capacity is fixed and can't scale in seconds. How do you keep a spike from either dropping requests or requiring you to over-provision hugely?
Spiky demand, fixed GPU capacity that can't scale instantly. How do you cope?
GPUs are extremely expensive; provisioning for peak wastes enormous money sitting idle most of the time, and you'll still be caught by a bigger-than-expected spike. A queue lets you provision for sustainable throughput and absorb bursts.
Dropping paid requests during your busiest moments is the worst outcome. Buffer them instead — users happily wait a bit for a generated image.
The queue absorbs bursts: jobs pile up and drain as GPUs free up, so a spike becomes a longer wait rather than dropped requests or wasted idle capacity. You provision for sustainable throughput, and the queue handles the peaks.
Put a job queue between the API and a pool of GPU workers. The API enqueues; workers pull jobs at their own sustainable rate. A demand spike simply lengthens the queue (and the ETA) instead of dropping requests or forcing you to over-provision GPUs for peak. You size the fleet for steady throughput; the queue absorbs the bursts. This is the shock absorber that makes fixed, costly capacity workable.
Why this piece earns its place
A queue absorbs a burst, but it does not have unlimited useful depth. Past some backlog the honest answer changes: a job accepted with a wait longer than the user will stay is worse than a refusal, because they are gone and you still spend the GPU seconds rendering for nobody. So the door needs a policy, and the number it reads is the time the current fleet needs to drain what is already queued, not a raw count of jobs — past that line, decline quickly or offer a cheaper variant, while the backlog still looks perfectly healthy on a graph. The second thing a queue changes is delivery semantics. A worker can die mid-denoise, and unless you have deliberately written the half-finished latent somewhere, none of that work survives: whatever the worker pulled becomes visible again to the fleet once its lease expires, and the next worker starts the generation from noise. So a job runs at least once and occasionally twice. Design the downstream write for that now — key the stored image by job id so a rerun overwrites rather than leaving two images the status record has to choose between, and write the image before marking the job done. Reverse that order and a crash in the gap gives you a job marked finished with nothing to serve.
- queueabsorbs spikes
- pullworkers at their rate
- size for throughputnot peak
What the new pieces do
- Job Queuebus
- Holds pending generation jobs, decoupling bursty demand from the fixed, expensive GPU fleet. Absorbs spikes.
- GPU Workerservice
- Pulls a job, runs the iterative denoising loop on a GPU (batched with others), and writes the finished image out.
- Job Statusdata
- Tracks each job's state and final image URL, so the API can answer "is it ready?" without holding the connection.
Step 4 · Inside the box
The diffusion inference loop
A worker has a job — now it actually generates the image. What is the model doing, and why is it so compute-heavy?
The worker runs the diffusion loop: start from random noise in a compressed latent space, then iteratively denoise it over many steps, at each step nudging it toward the text prompt (via cross-attention on the prompt embedding, with a guidance scale controlling how strongly). After the final step, a decoder turns the clean latent into pixels. Each of the dozens of steps is a full neural-network pass — which is why it's slow and GPU-bound. Fewer steps = faster but lower quality; that tradeoff is a key dial.
Why this piece earns its place
Reproducibility here is something you engineer, not something the loop hands you. The same prompt and seed only return the same picture if the scheduler, the weights version and the numeric precision are pinned alongside them — and once the next step batches jobs onto one GPU, batch shape and GPU model can change which kernels run and therefore the last bits of the image. Treat that as a design requirement rather than a pleasant property, because it is load-bearing: regenerating, sharing a seed and reproducing a complaint all depend on it, and it makes the weights version state rather than configuration. Record the weights version, the scheduler, the seed and the settings on every result, and ship new weights as a new version instead of an upgrade in place. A user who saved a seed, came back and got a different picture has lost something they thought was theirs, and nothing in your telemetry will report it — no error, no latency change, just a report you cannot reproduce because the model that made the original is gone. Which settles where the knobs live: steps and guidance in the request, where a caller can trade quality for wait; the model version in config, and on the result.
- noise → imageiterative
- ~20–50denoising steps
- guidanceprompt adherence dial
What the new pieces do
- Diffusion Modeldata
- The loaded model weights. Denoises a random latent toward the prompt over many steps, then decodes to pixels.
Step 5 · Feed the fleet
Batched, autoscaling GPU workers
GPUs are most efficient when kept busy with parallel work, they're expensive, and demand rises and falls. How do you run the worker fleet cost-effectively?
Batch multiple generation requests together on each GPU — GPUs are throughput machines, so processing several images in parallel dramatically raises utilization and lowers cost per image. Autoscale the worker pool on queue depth (more workers when the backlog grows, fewer when it drains) to track demand without paying for idle peak capacity. Keep model weights resident (loading a multi-GB model is slow) and manage cold starts by keeping a warm pool. Route different models/resolutions to appropriate GPU types.
Why this piece earns its place
Batching is not free, and the bill is paid in latency. You either hold a job while a batch fills, adding wait to every request in order to help the ones behind it, or you launch half-empty batches and give away the throughput you batched for. That fill window is the real knob, and it belongs per model, because the right value depends on arrival rate rather than on hardware. A batch is also a unit. Jobs in it finish together, so a twenty-step request sharing a batch with a fifty-step one waits for the fifty — batch by shape, matching model, resolution and step count, or your fast path silently inherits the slow path's latency while each job's own duration still reads as normal. The same unit rule governs failure. Resident weights are a fixed floor; everything above them scales with batch size times the pixel count, so doubling an image's width and height is roughly four times the working set, and attention over the latent grid grows faster still. An out-of-memory kills every job in the batch, not just the one that crossed the line, so retry the survivors alone. And queue depth is the right signal to scale on, but it counts jobs that are not equal — pair it with how long the oldest job has waited, because a new node boots and then pulls multi-gigabyte weights before it takes anything.
- batchparallel on GPU
- autoscaleon queue depth
- warm weightsavoid cold start
Step 6 · Store and deliver
Object store + CDN
The worker produced an image. Where does it go, and how does the user (and anyone who views it later) get it fast and cheaply — without shuttling image bytes back through your job system?
Write the finished image to object storage and record its URL in the job result — the API returns that URL rather than passing image bytes around. Serve the image through a CDN so the user (and any subsequent views, shares, or galleries) fetch it from a nearby edge, cheaply and fast. Generated images are immutable and often re-viewed, making them ideal CDN content. Store thumbnails/variants too if the product needs them.
Why this piece earns its place
The URL is the permission. Write generations into a public bucket and every one of them is readable by anyone holding the link, which includes anyone handed it by a log line, a referrer header or a screenshot. Private output needs signed URLs, and that immediately fights the CDN: a signature scoped to one viewer is a different URL per viewer, so the edge caches nothing and you pay origin egress on every view. The ways out are all deliberate — sign at the edge, carry the signature in a cookie instead of the path, or accept that only public galleries get cached — and choosing late means changing how every image in the product is addressed. The other thing storage does is never shrink. Generations accumulate, and unlike GPU seconds this bill arrives every month forever, so anything not explicitly kept needs an expiry. That turns the URL in the job record into a claim rather than a guarantee: it can be gone a month later, so the record should carry the lifetime and the client should be told the image expired instead of being handed a dead link. Keep that lifetime short enough that a takedown you cannot confirm reached every edge still ends on its own.
- object storedurable images
- URLnot bytes in the API
- CDNfast, cheap delivery
What the new pieces do
- Image Storedata
- Durable object storage for finished images; the job result points here rather than passing image bytes around.
- CDNcache
- Serves the generated image to the user (and any future views) from a nearby edge, cheaply and fast.
Step 7 · Both ends of safety
Prompt and image moderation
Image generation is uniquely risky: a benign-looking prompt can produce harmful, illegal, or NSFW imagery, and users will deliberately try. Filtering text alone is not enough — you have to check what the model actually drew. How do you make it safe?
A harmless-looking prompt can still yield harmful imagery. What's required?
You screen the prompt before generation (block obvious policy violations, prompt tricks) AND run the generated image through visual moderation (NSFW/CSAM/violence classifiers) before delivery — because the model can produce disallowed content from prompts that look innocuous. Text filtering alone can't see the pixels.
Prompt-only filtering misses the core risk: the image itself. Innocuous or obfuscated prompts can still yield harmful pixels, so you must moderate the output image, not just the input text.
For image generation, after-the-fact moderation is far too late and legally unacceptable for the worst content. You must gate both input and output proactively, before delivery.
Moderate on both ends. Input: screen the prompt (policy violations, known evasion tricks) before spending a GPU on it. Output: run every generated image through visual moderation (NSFW / violence / CSAM classifiers) before it's ever delivered — because the model can draw disallowed content from prompts that look harmless. Block, blur, or reject on a hit, and log for review. Add provenance (watermarking / C2PA) so generated images are identifiable. Safety here is mandatory and defense-in-depth.
Why this piece earns its place
Every one of these checks is a threshold, and its two ways of being wrong do not cost the same. A false positive is a paying user who waited, received nothing, and now wants an explanation. A false negative in the worst categories has no ceiling at all. So the cutoff is set per category rather than once for the whole service — severe where the downside is unbounded, looser where the cost is an irritated customer — and you decide explicitly whether a blocked generation is billed, because those GPU seconds were genuinely spent and somebody will ask. The harder problem is that the failure that matters cannot be measured from production data: blocks are counted, but what slipped through is by definition unlabelled, so the miss rate never reaches a dashboard and the graph looks calm either way. That number comes from sampled human review and deliberate red-teaming against the current weights, repeated whenever the model or the classifier changes. Two smaller consequences. The output check sits on the critical path behind the slowest stage in the design, so it has to be a small model rather than a second large one. And the evidence you retain for review is the very material you are containing — restrict who can open it, and give it a retention limit.
- inputscreen the prompt
- outputscreen the image
- provenancewatermark / C2PA
What the new pieces do
- Safetyservice
- Moderates the prompt before generation and the image after — both are required, since the model can draw what text filters miss.
Step 8 · The sharp edges
Cost, priority & cold starts
The realities of running expensive generative infra: GPU cost dominates everything, free and paid users compete for the same fleet, latency varies, and identical prompts get regenerated.
Control cost aggressively — GPU time is the bill — with batching, right-sized GPUs, step-count tuning, and per-user quotas. Prioritize the queue (paid/interactive jobs jump ahead of free/batch) so premium users get low latency while free traffic fills spare capacity. Manage cold starts with warm pools and by keeping popular models resident. Cache/dedupe identical prompt+seed requests. Set honest ETAs and stream progress (or intermediate previews) so a multi-second wait feels responsive. And handle failures with idempotent retries that don't double-charge GPU time.
Why this piece earns its place
A strict priority queue starves. If paid work arrives continuously, free jobs are never at the front, and the queue meant to hold them becomes the place that work goes to die — no error anywhere, just a class of users whose images never come. The fixes are aging, where a job's effective priority climbs with the time it has already waited, or reserving a slice of the fleet for the lower class outright; reservation is easier to reason about and to explain to the person paying for it. Preemption is the other half, and it is a choice rather than a constraint. The loop's live state is one latent plus a step index — the same intermediates you decode for previews — so a running job can be checkpointed and picked up elsewhere. What that costs is moving tensors off the GPU and a guarantee that the receiving worker holds identical weights and scheduler, which is why most fleets skip it and preempt only at job boundaries. Either way, say which you chose, because it sets the promise: at job boundaries, an urgent job arriving at a full fleet waits a whole generation at best. And none of these levers work unless the job carries its own cost — tag it at submit with user, model and the settings that drive GPU time, or the quota you enforce is a guess about a resource you never measured.
You did it
You just designed a text-to-image service.
Everything you assembled, in order
- Synchronous generation fails: jobs are slow and GPU-bound, so you can't hold a request or absorb spikes.
- Make it async: accept fast (job id), run behind the scenes, let the client poll/subscribe for the result.
- A job queue between API and GPU workers absorbs bursts and lets you size for throughput, not peak.
- The worker runs the diffusion loop — iterative denoising in latent space toward the prompt.
- Batch requests on each GPU and autoscale the fleet on queue depth; keep weights warm.
- Write images to object storage and deliver via CDN — pass a URL, never bytes through the API.
- Moderate both the prompt and the generated image; add provenance. Control GPU cost and prioritize the queue.