The whole design, in writing
Learn AI system design by building a text-to-speech / voice synthesis service like ElevenLabs, Play.ht or a hosted TTS step by step. An interactive guide covering the synthesis pipeline (text normalization, acoustic model, vocoder), batch vs streaming synthesis, first-audio latency for voice agents, voices and prosody control, caching common phrases + CDN, GPU scaling, voice cloning safety, and the unhappy paths.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What makes text-to-speech hard?
Turning text into a natural voice has two faces. Batch: narrate a whole article or audiobook — quality matters, latency doesn't. Streaming: a voice agent that must start speaking within ~200ms of deciding what to say, or the conversation feels broken. And "natural" hides real complexity: pronouncing "Dr.", "$5", and "2024" correctly, with human-like rhythm and emotion.
A text-to-speech (TTS) service synthesizes speech through a pipeline — text normalization → acoustic model → vocoder — served in batch or streaming mode. The hard parts: getting the front-end (pronunciation) right, hitting the first-audio latency budget for interactive agents, and controlling GPU cost with caching.
What the new pieces do
- Textclient
- The text to speak, a chosen voice, and optional prosody controls (SSML). May be a whole document (batch) or a live stream (agent).
Step 1 · "Natural" is the hard part
More than reading letters
It's tempting to think TTS is "map each letter to a sound." Why is producing a convincing voice actually much harder than that?
Why is natural TTS much harder than mapping letters to sounds?
"$5 on 3/4" must become "five dollars on March fourth", homographs need context ("read" past vs present), and the voice needs natural rhythm, stress and emotion — not a flat letter-by-letter readout. TTS is a multi-stage problem: normalize → pronounce → make it sound human.
File size is trivial. The difficulty is linguistic and acoustic: correctly interpreting text into spoken words and rendering natural, expressive prosody — a flat phoneme mapping sounds robotic and mispronounces real text.
They don't — that's exactly the problem. Context decides pronunciation (homographs), and raw text is full of numbers, symbols and abbreviations that must be expanded to words first.
Natural speech is a multi-stage problem. You must normalize text ($5 → "five dollars", Dr. → "Doctor", 3/4 → "March fourth"), resolve pronunciation including homographs from context ("read" past vs present), and render human-like prosody — rhythm, stress, intonation, emotion. A flat letter-to-sound map is robotic and wrong. This is why TTS is a pipeline, not a lookup.
Why this piece earns its place
Architecturally the front-end is the one stage you can change without training anything, and that is what makes it the cheap half of the page. A mispronounced product name is a lexicon entry that ships this afternoon; the same complaint about the voice itself is a training run and a re-evaluation. Adding a locale is mostly front-end work for the same reason — ordinals, currency, dates and address formats are local convention — long before it is a second acoustic model. What surprises people is that normalization is not decidable from the text alone. 3/4 is March fourth to one caller and the fourth of March to another; Dr. is Doctor before a name and Drive after one; St is Saint or Street. The front-end cannot read that out of the string, so the ambiguity belongs in the request — locale, plus a hint about what kind of text this is — rather than in a global default that is quietly wrong for half your customers. The one time you can skip most of this is when you own the text upstream: if it comes from templates you generate, emit spelled-out words yourself and hand the service something already spoken.
Step 2 · The front door
A gateway for batch and streaming
Like ASR, TTS serves two modes with different SLAs. What does the entry point do when a synthesis request arrives?
A gateway accepts the text, chosen voice, and optional SSML/prosody controls, checks the cache, and routes by mode. Batch: synthesize the whole text and return an audio file (URL). Streaming: begin emitting audio as soon as the first chunk is ready and stream it out over a persistent connection — essential for voice agents where first-audio latency is the metric that matters. Both drive the same synthesis pipeline and GPU fleet.
Why this piece earns its place
The two modes differ in connection shape, not only in latency target, and that is what makes the gateway real work rather than a router. A batch request behaves like any request: it arrives, it is served, the socket closes, and capacity is requests per second. A streaming request holds a connection open for as long as the utterance lasts, so the gateway's actual capacity is concurrent open streams — and an autoscaler watching request rate will report comfortable headroom while every worker is pinned. Deploys stop being routine for the same reason: you cannot drain a node with live streams the way you drain a request server, so a rolling restart cuts people off mid-sentence unless the drain refuses new streams and waits out the old ones. Retries split along the same line. A batch caller is waiting on a URL, so a retry is safe once the request carries an idempotency key and the second attempt lands on the first result rather than on a second take. Second take is the exact phrase: expressive models sample as they generate, so the same inputs return the same audio only when a seed is pinned — pin one on anything you intend to store, cache or re-serve. A streaming call has no retry at all, because the caller has already played the first half; the honest recoveries are resume-from-offset or stop.
- batchwhole file (URL)
- streamingfirst-audio fast
- SSML / voicecontrol inputs
What the new pieces do
- Gatewaygateway
- Accepts synthesis requests, checks the cache, and routes to the pipeline — streaming audio out or returning a file.
Step 3 · Speak before it's all written
Streaming synthesis
A voice agent has decided to say a two-sentence reply. Waiting to synthesize the entire thing before playing any of it adds seconds of dead air. How do you make it feel instant?
A voice agent must reply fast. How do you avoid seconds of silence before it speaks?
Synthesizing everything first adds latency proportional to the reply length before any sound — dead air that breaks conversational flow. You want to start speaking as soon as the first chunk is ready.
Speaking rate doesn't address the initial delay before audio starts. The fix is to generate and stream the first chunk immediately, continuing to synthesize the rest as it plays.
Chunk the text (by clause/sentence), synthesize the first chunk fast, and start streaming audio while later chunks are still being generated. First-audio latency drops to the time for one small chunk, and playback stays ahead of generation, so the reply begins almost instantly and continues seamlessly.
Stream the synthesis. Chunk the text (by clause/sentence), synthesize the first chunk fast, and begin streaming its audio immediately while later chunks are generated in the background — playback stays just ahead of generation. First-audio latency collapses from "synthesize the whole reply" down to "synthesize one small chunk", so a voice agent starts speaking almost instantly and continues seamlessly.
Why this piece earns its place
Chunking looks like a text-splitting detail, and the awkward part is that the text is itself arriving as a stream. The model upstream is still writing the sentence you are trying to cut, so the chunker commits to a boundary before it has seen what follows — wait for the sentence to finish and you have put back the delay you just removed, cut at the first comma and you commit to a break the rest of the clause would have argued against. The boundary itself is a real choice, not a detail. Synthesize each chunk knowing nothing of the last and you can generate several at once, but the acoustic model starts a fresh pitch contour every time, so the back half of a sentence begins as though it were a new sentence and a listener hears the seam. Carry the previous chunk's state forward and it stays one utterance, at the price of serializing generation: chunk two cannot start until chunk one has finished. Make that trade deliberately. And the failure to design against is not a slow start, it is a stall — if generation falls behind the player you get a gap mid-word, which a listener hears as broken rather than slow, so track the lead in seconds of buffered audio and widen chunks before it runs out.
- ~200msfirst-audio target
- chunksynth incrementally
- play aheadof generation
Step 4 · Inside the pipeline
Normalize → acoustic → vocoder
Concretely, how does normalized text become a waveform? What are the stages and why are they usually split?
Three stages. (1) Text front-end: normalize (numbers/dates/abbreviations → words) and convert to phonemes (grapheme-to-phoneme), with prosody from SSML/context. (2) Acoustic model: turn the phonemes + target voice + prosody into a mel spectrogram (how it should sound over time). (3) Vocoder: convert the mel spectrogram into an actual waveform. Splitting acoustic model from vocoder lets each be optimized (and swapped) independently; some modern systems fuse them end-to-end.
Why this piece earns its place
The seam between the two back-end stages is a contract, and it is written in numbers that rarely reach the design doc: sample rate, hop length, number of mel bins, the frequency range and the scale. An acoustic model and a vocoder are interchangeable only inside one of those conventions. Pair two that disagree and nothing throws — you get audio, and it is metallic, or pitched wrong, or slightly too fast. So the mel specification is part of the model contract, pinned beside both checkpoints, and swapping the vocoder independently is true only within it. That seam is also the debugger. When a sample comes out bad you can dump the mel, render it with a known-good reference vocoder and listen: still wrong means the acoustic model, fine means the vocoder. Losing that single inspection point is what fusing the stages actually costs you — not quality, but the ability to say which half broke at three in the morning. And size both halves in seconds of audio rather than in requests or characters: the acoustic model is decoding mel frames and the vocoder is rendering them, so what a request costs is set by how long its result takes to say, and one narration can outweigh a long tail of notification clips on the same dashboard.
- front-endtext → phonemes
- acousticphonemes → mel
- vocodermel → waveform
What the new pieces do
- Text Front-endservice
- Normalizes text (numbers, dates, abbreviations, currency) into spoken words and maps to phonemes — the top error source.
- Acoustic Modelservice
- Turns the normalized text/phonemes (with the target voice + prosody) into a mel spectrogram — how it should sound.
- Vocoderservice
- Converts the mel spectrogram into an actual audio waveform — the second stage that produces the sound you hear.
Step 5 · Voices & expression
Voices, cloning & prosody control
Users want a specific voice, sometimes their own, and control over how it sounds — emphasis, pauses, emotion, speed. How does the service offer that?
A user uploads a 10-second audio clip and asks to clone that voice. What should the service require before generating anything in it?
Possessing a recording of someone's voice (a public speech, a voicemail, a video clip) is not consent to clone it — this is exactly the gap that enables impersonation and fraud.
Consent has to be structural, not assumed: verify the person cloning a voice is authorized to, gate the resulting voice behind that check, and watermark its output so misuse is detectable downstream.
A label after the damage is done doesn't prevent the fraudulent call or the fake endorsement from being generated and sent in the first place — the gate belongs before generation, not as a disclaimer on the result.
Support multiple voices as speaker embeddings/models the acoustic stage conditions on, plus voice cloning (derive a new voice embedding from a short sample). Give control via SSML or parameters — emphasis, pauses, rate, pitch, emotion/style — so callers shape the prosody. Because cloning is powerful and abusable, gate it behind consent and verification, and add audio watermarking so synthetic speech is detectable. Voice is both the product and a safety surface.
Why this piece earns its place
A voice is not a setting, it is a stored artifact with an owner, and owners change their minds. Consent gets withdrawn, a clone gets disputed, a licensed voice reaches the end of its term — and each of those is a revocation, which is far harder than a refusal. Deleting the embedding stops new synthesis and does nothing about audio already rendered in that voice, sitting in object storage and cached at the edge, which is the material anyone would actually play back. So the voice id has to travel with the output, in the cache key and in the stored file's metadata, or a takedown arrives and you cannot even enumerate what to purge. The same artifact view explains why upgrading the acoustic model is never merely an upgrade: the new one renders the same voice slightly differently, so a customer who published a narration and comes back to re-render one corrected sentence gets a sentence that does not match the chapter around it. Voices get pinned versions, and old versions stay servable longer than you would like.
- voicesspeaker embeddings
- cloningfrom a sample (gated)
- SSMLprosody control
Step 6 · Don't re-synthesize the world
Caching & delivery
Enormous fractions of TTS traffic are identical — the same IVR prompts, notifications, and stock phrases synthesized millions of times in the same voice. Re-running the GPU for each is pure waste. And finished audio needs fast delivery. How?
Cache synthesized audio keyed by (text + voice + params): a repeated phrase is served from the cache and never touches the GPU — a massive cost and latency win for the common, static content that dominates many workloads. Store generated audio in object storage and deliver via CDN (audio is immutable and often re-played, ideal edge content). The result is a URL; only genuinely novel text pays for synthesis.
Why this piece earns its place
One hit rate across the whole service is a number that describes nobody. Traffic splits into classes that behave in opposite directions: prompts and notifications repeat almost totally, while a voice agent's replies are nearly all unique, since the point of them is that they were composed for this conversation. Average the two and you get a healthy-looking figure that hides GPU load being set entirely by the agent traffic. Size the fleet on the miss rate of the class that misses, and report hit rate per class or not at all. The same split distorts latency: with a cache in front of a GPU the distribution is bimodal, so the median measures the cache and synthesis only shows up in the tail — an SLO written on p50 stays green through a real regression in the thing you built. Two smaller facts worth carrying. The cache is bounded in bytes, not entries, since audio is enormous next to the text that keys it: one long clip evicts many short ones, and a policy that counts entries will thrash. And key it on the front-end's output rather than the raw string — two callers who punctuate the same sentence differently, or spell a number out, want byte-identical audio and would otherwise miss each other, at the cost of normalizing before the lookup, which is nothing against a miss.
- cachetext+voice+params
- hitskip the GPU
- CDNdeliver static audio
What the new pieces do
- Phrase Cachecache
- Caches synthesized audio keyed by (text + voice + params), so common/repeated phrases skip the GPU entirely.
- Audio Storedata
- Durable storage for generated audio; the result is a URL, and static audio is delivered via CDN.
- CDNcache
- Serves finished, immutable audio from a nearby edge for fast, cheap playback and re-listens.
Step 7 · Feed the fleet
GPU scaling
Novel synthesis is GPU-bound (the vocoder especially), demand varies, and streaming has a tight latency budget. How do you run the fleet economically without breaking real-time?
Same GPU-economics as other inference services: batch synthesis requests for throughput on the batch path, but keep streaming batches small so first-audio latency stays low. Autoscale on load, keep models warm (avoid cold starts loading multi-GB weights), and use faster/lighter models or vocoders for the streaming path where latency dominates. Combined with the cache (most traffic never reaches a GPU), this keeps cost in check while protecting the interactive SLA.
Why this piece earns its place
One pipeline does not have to mean one pool. What the two paths gain by sharing is the pipeline itself — the same models, the same code path, one deploy, so the modes cannot quietly drift apart — and none of that requires them to share a device. Sharing devices costs the interactive SLO: a large batch occupying a GPU adds queueing delay to whatever streaming request lands beside it, and that surfaces as first-audio jitter rather than as an error, so it survives every dashboard you have. Partition the fleet, keep one pipeline across both partitions, and treat the streaming pool's lower utilization as the price of the SLO rather than waste to reclaim. Autoscaling then has a timing problem worth naming: a GPU is not ready when the instance is, it is ready when the weights are resident, and how long that takes is a property of how you package and place them rather than of the traffic that caused it. Spikes will not wait for it, so autoscaling covers the daily curve and a warm pool covers the spike. Under real overload, queue depth is the wrong lever. A streaming request that waits has already failed, because its audio lands after the conversation moved on, so requests carry a deadline and the gateway refuses what it cannot meet. The batch path, having no deadline, is what belongs on preemptible capacity.
- batchthroughput path
- small batchstreaming latency
- warm + autoscalecost
Step 8 · The sharp edges
Pronunciation, latency & abuse
TTS fails in fluent, confident ways: the front-end mispronounces text, homographs go wrong, streaming latency slips, and voice cloning invites impersonation and fraud.
Harden the front-end — robust normalization for numbers/dates/currency/units, a pronunciation lexicon for names and jargon, and context-aware homograph resolution — since these errors are fluent and confident, not crashes. Protect the latency budget with chunked streaming, low-latency models, and caching. Combat abuse: consent/verification for cloning, watermark synthetic audio, rate-limit, and moderate text (don't voice disallowed content). Fail gracefully — fall back to a simpler voice/model rather than silence.
Why this piece earns its place
The failures on the other steps announce themselves — a stall leaves a gap mid-word, a missed deadline comes back as a refusal, a cold pool shows up as queueing. A wrong word does not, so the signal for this step has to be manufactured rather than monitored. In practice that means a held-out set of deliberately hard text — names, units, addresses, currency, mixed-language strings — re-rendered on every model or lexicon change and compared against audio someone already approved, plus a route for a customer to report a single wrong word. That route is the part built too late. Overrides are also mostly per tenant rather than global: a company's product name, a hospital's drug list, a game's invented places. If the lexicon is an engineering backlog item then every customer waits on your release train for a word only they say, so make it data they configure, with a test attached to each entry. And treat watermarking as a system rather than a feature: a mark is worth exactly what the detector can still read after the audio has been re-encoded and resampled on its way to wherever it gets played, and that detector is a service you staff and run with no revenue against it.
You did it
You just designed a text-to-speech service.
Everything you assembled, in order
- Natural TTS is a pipeline, not a lookup: normalize text, resolve pronunciation, render human prosody.
- A gateway serves batch (whole file) and streaming (first-audio-fast) over the same synthesis pipeline.
- Streaming synthesizes in chunks and plays the first audio immediately — first-audio latency is the metric.
- The pipeline is text front-end → acoustic model (mel) → vocoder (waveform), often split for optimization.
- Voices are conditioning inputs; cloning is powerful and must be gated by consent + watermarking.
- Cache synthesized audio by text+voice+params so repeated phrases skip the GPU; deliver via CDN.
- Scale GPUs with batching (small for streaming) + autoscaling + warm weights; harden the front-end.