The whole design, in writing
Learn AI system design by building a speech-to-text / ASR service like Whisper-at-scale, Deepgram or Google Speech step by step. An interactive guide covering batch vs streaming transcription, the ASR inference pipeline, chunking and voice activity detection, a batched autoscaling GPU fleet, post-processing (punctuation, diarization, timestamps), and the unhappy paths (real-time latency budget, accuracy vs chunk size, noisy audio, cost).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What makes speech-to-text hard?
Turning audio into text splits into two very different products. Batch: transcribe an uploaded hour-long recording accurately (podcasts, meetings) — accuracy matters, latency doesn't. Streaming: caption a live conversation as it's spoken, showing words within a fraction of a second — latency is everything. Same model, opposite constraints, and both run heavy GPU inference.
A speech-to-text (ASR) service serves both modes over a GPU model. The shape: a gateway that splits streaming from batch, a batched GPU worker fleet running the model, chunking/VAD to feed it, and post-processing (punctuation, timestamps, diarization) to make raw tokens useful. The hard parts are the streaming latency budget and cost.
What the new pieces do
- Audio Sourceclient
- A live microphone stream (real-time captions) or an uploaded recording (batch transcription). Two very different modes.
Step 1 · Two modes, one model
Batch vs streaming
Before any architecture, recognize that "transcribe audio" is really two problems with opposite priorities. What are they, and why can't one design serve both naively?
Why can't a single naive "upload → transcribe → return" design serve all speech-to-text?
Batch can wait and see the whole recording for max accuracy; streaming must emit words within a fraction of a second before the utterance is even finished. These opposite priorities need different handling (async job vs persistent low-latency connection with partial decoding), even if they share the model.
File size is manageable. The real split is latency: real-time captioning can't wait for a whole-file transcription, so it needs incremental, low-latency processing that a simple upload-and-return can't provide.
The same model can serve both; the difference is how you feed it and return results — full-file async for batch, chunked low-latency streaming for real-time.
Two modes, opposite constraints. Batch can wait and use the whole recording for maximum accuracy — an async job. Streaming must emit words as they're spoken, within a fraction of a second, before the sentence is even done — a persistent connection with incremental, low-latency decoding. They share the model but need different plumbing, so the design splits them at the front door.
Why this piece earns its place
The split is easy to agree with and hard to hold, because the two modes are also measured differently and most teams only measure one. A batch transcript is scored against a reference after the fact, on a finished file. Streaming has a second axis batch does not have at all: how long a word takes to appear after it was spoken, and how much a partial changes before it settles. A streaming service can produce an excellent final transcript and still feel broken — flickering and rewriting itself under the reader — and a word-error number will never show it. The split also changes the API contract. Batch returns one immutable result; streaming returns a sequence of revisions, so every consumer downstream has to be able to unwrite text it has already displayed. A caption renderer or a log sink that only appends will duplicate and corrupt the transcript, and that is a client bug you will be blamed for. If the product only ever ingests recordings — a meeting archive with no live view — skip the streaming plumbing entirely. It is the expensive half.
Step 2 · The front door
A gateway that splits the paths
Given two modes, what does the entry point do the moment audio (live or uploaded) arrives?
A gateway routes by mode. For streaming, it holds a persistent connection (WebSocket/gRPC) and pipes audio frames toward the workers, returning partial transcripts live. For batch, it accepts the file, enqueues an async job, and returns a job id to poll or a callback URL. Both feed the same GPU model fleet, but through different flow-control: real-time streaming vs buffered async jobs.
Why this piece earns its place
A stateless endpoint can hand any request to whichever worker is free. A streaming connection cannot: the decoder carries context from one chunk to the next, so a call is pinned to the worker it started on for its whole life. The gateway is what holds that call together, and the settings it holds — sample rate and encoding, a language hint, the vocabulary for this conversation — are agreed once at connection open and cannot be changed halfway through a sentence. The queue and the transcript store keep state too, but theirs survives a restart; this state is live, and it dies with the socket. Two things fall out of that. Clients drop, especially mobile ones, so the gateway needs a resume token and a defined behaviour for the audio that arrived while nobody was listening, or the caller silently loses the tail of a sentence and never learns it happened. And the connection is where you decide what a partial means: whether a revision may rewrite anything already sent, or only the last unsettled segment. Publish that rule, because callers will build against whichever one you happen to ship.
- streamingpersistent conn, partials
- batchasync job + callback
- shared fleetone model, two paths
What the new pieces do
- Gatewaygateway
- Splits traffic: a persistent connection for streaming audio, or an async upload endpoint for batch files.
Step 3 · Buffer the batch
Async jobs for uploads
Uploaded recordings are heavy (an hour of audio is minutes of GPU work) and arrive in unpredictable bursts. How do you handle them without dropping jobs or starving the latency-sensitive streaming path?
Send batch uploads through a job queue to the GPU workers, exactly like any heavy async job: enqueue, transcribe at the fleet's sustainable rate, deliver via poll or callback. Crucially, keep batch and streaming on separate paths/queues (or reserved capacity) so a flood of uploaded files can't consume the GPUs that real-time captioning depends on. Batch absorbs bursts as backlog; streaming keeps its low-latency lane.
Why this piece earns its place
A transcription job is long, and long jobs make the queue's delivery guarantees matter. At-least-once means a worker that dies near the end of an hour of audio redoes all of it unless the job checkpoints per segment, and that is GPU time bought twice for one transcript. It also means the completion can fire more than once, which costs no compute at all but arrives as a duplicate callback — and as a second charge, if billing meters completion events rather than audio processed. So jobs want an idempotency key, per-segment checkpoints, and a callback the receiver can safely get twice. Queue depth is the signal to scale on, but the unit is not jobs: one job is a voicemail and the next is a three-hour lecture, so count the backlog in seconds of audio and the same number becomes an honest ETA and a measure of the work rather than of the paperwork. One FIFO queue also stops being enough quickly: a bulk import from a single customer will sit in front of everybody else's short files for hours. Split by expected duration, or schedule per tenant. If uploads are small and infrequent, none of this earns its keep — transcribe synchronously and return the text.
- batch → queueabsorb bursts
- isolatestreaming stays low-latency
- poll / callbackdeliver batch results
What the new pieces do
- Batch Queuebus
- Holds batch transcription jobs so uploaded recordings drain at GPU pace, isolated from the live streaming path.
- ASR Workerservice
- Turns audio features (a mel spectrogram) into text tokens on a GPU — batched with other requests for throughput.
Step 4 · Inside the model
The ASR inference pipeline
A worker has audio to transcribe — what does the model actually do to turn a waveform into words?
Convert the raw waveform into audio features — typically a mel spectrogram (a time-frequency representation) — then feed it to the ASR model (a Whisper-style encoder-decoder or transducer), which produces a sequence of text tokens, decoded (often with beam search and an optional language model) into words. It's a full neural-network inference per segment of audio, which is why it's GPU-bound. Modern models also do language ID and are robust to accents/noise as part of the same pass.
Why this piece earns its place
The decoder's cost tracks the tokens it emits, not only the length of the audio, and that asymmetry is what needs guarding here. Hand a sequence model a segment with no speech in it and it will emit confident text, and can repeat itself until something stops it — a quality bug and a bill at once — so cap tokens per segment and treat a segment that hits the cap as suspect. Word-level timing is the other thing to settle before anyone promises it. A Whisper-style encoder-decoder emits coarse segment timestamps; per-word times come from a separate alignment pass, either dynamic time warping over the decoder's cross-attention or an external aligner. A CTC or transducer model aligns frame by frame and gives them away. So if captions or word-level search are in the requirements, that belongs in the argument at model-selection time, not downstream where it looks like a formatting detail. Placement matters at the front too: resampling, downmixing and the spectrogram are ordinary signal processing, small beside the forward pass, and running them on the accelerator mostly spends GPU the decoder could have used. Store the audio as it arrived, not the normalized copy, or every future re-run inherits today's preprocessing decisions.
- mel spectrogramaudio features
- seq2seqaudio → tokens
- beam searchdecode to words
What the new pieces do
- ASR Modeldata
- The loaded acoustic/sequence model (Whisper-style). Maps audio features to a token sequence.
Step 5 · Feeding the model
Chunking & voice activity detection
You can't feed an open-ended live stream (or a one-hour file) to the model in one shot — models have a bounded input window, and streaming needs to emit results before the audio ends. How do you segment the audio well?
You can't feed unbounded/live audio to a fixed-window model in one shot. How do you segment it?
Hard fixed cuts slice words in half and lose the context that spans a boundary, hurting accuracy. You want boundaries at natural pauses and overlapping windows so no word is cut and context carries across.
For batch that's fine, but for streaming you must emit partial results as speech happens — you can't wait for the utterance to end. Streaming requires incremental chunked decoding.
VAD finds speech vs silence so you can segment at natural pauses (not mid-word), overlap windows so context spans boundaries, and — for streaming — decode each chunk as it arrives, emitting partial transcripts that may be refined as more audio comes in. This balances the model's bounded window against accuracy and latency.
Use voice-activity detection (VAD) to split audio into speech segments at natural pauses (not mid-word), with overlapping windows so context carries across boundaries. For streaming, decode each chunk incrementally as it arrives and emit a partial transcript that can be revised as more audio comes in; finalize when the segment ends. Chunk size is the core dial: smaller = lower latency but less context (lower accuracy), larger = the reverse.
Why this piece earns its place
Overlap is the right call, and it has a consequence: the same speech is decoded twice, and the two decodes will not agree. You need an explicit stitching rule — where the seam falls, which side wins — or the transcript repeats a word or two at every boundary. That defect hides completely if you measure accuracy per chunk instead of on the assembled transcript, which is the natural way to build the harness and the wrong one. VAD is a cost lever as well as an accuracy one, since audio with no speech in it is audio you never pay to decode, though what that saves depends on the traffic: one participant's own microphone is mostly silence, a mixed-channel meeting recording is not. The danger runs the other way too — a VAD tuned tight clips the quiet start of a word, so the failure arrives as missing first words rather than as an error anywhere. Chunk size belongs in the request, not the config, because live captions and dictation want opposite settings from the same service. And when the client already knows where an utterance ends, as with push-to-talk, it should say so, and you can skip detection for that stream.
- VADsplit at pauses
- overlapkeep context
- partialsemit then refine
What the new pieces do
- VAD / Chunkerservice
- Voice-activity detection splits audio into speech chunks (overlapping windows) so the model gets bounded, meaningful segments.
Step 6 · Feed the fleet
Batched, autoscaling GPU workers
ASR inference is GPU-heavy, demand fluctuates, and you're paying for expensive accelerators. How do you run the worker fleet economically — while respecting streaming's latency?
The same GPU fleet serves both batch and streaming. Should both paths use the same batch size for inference?
A single batch size forces a bad compromise: large enough for batch throughput means streaming waits for a batch to fill (killing latency), small enough for streaming means batch wastes GPU efficiency it doesn't need to waste.
Batching raises throughput but adds latency by design (waiting to fill the batch). Batch jobs are latency-tolerant so they can use large batches for cost efficiency; streaming needs tiny, immediately-flushed batches to keep the sub-second budget, accepting lower GPU efficiency as the price.
Never batching throws away most of a GPU's throughput advantage for the batch path, where nothing is waiting on a fast response — that's the one place batching is nearly free.
Batch multiple requests per GPU to raise utilization (with care: streaming needs small, fast batches to stay low-latency, while batch jobs can use large batches for throughput). Autoscale the fleet on load (queue depth for batch, active-stream count for real-time). Keep the multi-GB model resident (warm pool, avoid cold starts), and consider smaller/faster models or quantization for the streaming path where latency dominates and a slight accuracy trade is acceptable.
Why this piece earns its place
Streaming capacity is not counted in requests per second, and once VAD is gating the decoder it is not counted in GPU time either. What a live call occupies is a slot: a pinned worker, the decoder state for that stream, and an open connection, held for as long as the call lasts. So the fleet has two numbers to plan against — compute, which follows speech, and concurrency, which follows connected streams — and a fleet sized on utilization alone will look half idle right up to the moment it runs out of slots. Scaling down is the hard direction: you cannot drain a worker by waiting for its in-flight work to finish when one item is a call that may run for another hour, so admission stops on that worker first and the rest is patience. Scaling up is not free either, because a new worker loads multi-gigabyte weights before it serves anyone, which is what the warm pool is for and why streaming headroom is provisioned ahead of need. Batch, being queued and checkpointed, is the half that can take interruptible capacity. Two practical notes: unequal-length batches pad every item out to the longest, so bucket by duration or pay to decode padding; and a weights rollout cannot swap under an open stream without the transcript changing style mid-sentence, so keep both versions resident and let existing calls finish where they started.
- batchthroughput (batch path)
- small batchlatency (streaming)
- autoscale + warmcost + no cold start
Step 7 · Make it useful
Post-processing
The model outputs raw tokens — often lower-cased, unpunctuated, with no idea who spoke or when. That's a wall of text, not a usable transcript. What turns tokens into a product?
Add a post-processing layer: punctuation & capitalization restoration, word-level timestamps (for captions/search), speaker diarization (who spoke which segment), language identification, and optional custom vocabulary / boosting for domain terms and names. This runs after (or alongside) decoding and is what makes a transcript readable, searchable, and attributable. It's a distinct value layer that can degrade gracefully if it fails.
Why this piece earns its place
Two different jobs share this one box. Punctuation, capitalization and the formatting of numbers and dates read the token stream and its timings, nothing else. Diarization and language ID read the waveform — voice activity, speaker embeddings, then clustering — so this stage wants the audio as well as the tokens, and that half is another neural pass over the sound, competing for the accelerators the previous step spends its time rationing. Post-processing is not the cheap tail of the pipeline it looks like; only its text half is. Formatting also breaks the timing map: three spoken tokens becoming one written 2023 collapses three timestamps into one, and captions and seek bars are driven by those timestamps, so decide what a formatted span's time means before someone builds on it. Recognition and diarization disagree about boundaries, too. Attach speakers per chunk and a turn that changes mid-segment is credited to whoever spoke most of it, producing a transcript that reads perfectly and is wrong about who said what; word-level attribution costs more and is what the feature is actually for. And the labels are anonymous — speaker one, speaker two. Turning them into names means enrolling voiceprints, which is biometric data in a way a transcript is not.
- punctuation+ capitalization
- timestamps + diarizationwhen + who
- custom vocabdomain terms
What the new pieces do
- Post-processservice
- Adds punctuation, capitalization, word timestamps, speaker labels (diarization) and language ID to raw tokens.
- Transcript Storedata
- Durable store of finished transcripts (and optionally the audio), returned to the client or via callback.
Step 8 · The sharp edges
Latency, noise & cost
Real ASR fights hard problems: the streaming latency budget, noisy/overlapping/accented audio, long-form recordings that exceed the model window, and GPU cost.
Hold the streaming latency budget with small chunks, fast/quantized models, and partial-then-refine output — accepting a small accuracy trade for responsiveness. Improve noisy audio with denoising/robust models and VAD, and handle overlapping speakers with diarization. Transcribe long-form audio by sliding the model window with overlap and stitching results. Control cost with batching, right-sized models per path, and caching/dedup of repeated audio. Fail gracefully — degrade post-processing before dropping the core transcript.
Why this piece earns its place
Word error rate is one number over a whole corpus, and the tokens that decide whether a transcript is usable — names, numbers, product terms — are a small minority of them, so a model can improve on average and get worse at exactly what a customer checks first. Keep separate evaluation sets per condition: phone audio, accents, background noise, each scored on its own, because a robustness gain on clean read speech will otherwise hide a regression on the traffic you actually serve. There is a versioning consequence too. Transcripts get compared: a customer who re-runs last month's recording and gets different words has found a bug as far as they are concerned, so record the model version beside every transcript and be able to serve a pinned one. And set the accuracy bar per product rather than buying the best model everywhere — a search index over recordings tolerates errors a legal or medical transcript never will, and that choice, not the GPU, is the real cost control.
You did it
You just designed a speech-to-text service.
Everything you assembled, in order
- Batch (accuracy, latency-tolerant) and streaming (real-time, low-latency) are opposite constraints — split them.
- A gateway routes streaming to a persistent connection and batch to an async job queue, sharing one GPU fleet.
- Keep batch and streaming on isolated paths so a batch flood can't starve real-time latency.
- The model turns a mel spectrogram into text tokens — GPU-bound seq2seq inference.
- VAD + overlapping chunks + incremental partials feed the fixed model window and trade latency vs accuracy.
- Batch on GPUs for throughput but keep streaming batches small; autoscale and keep weights warm.
- Post-processing (punctuation, timestamps, diarization, custom vocab) turns raw tokens into a usable transcript.
