The big idea
What makes speech-to-text hard?
Turning audio into text splits into two very different products. Batch: transcribe an uploaded hour-long recording accurately (podcasts, meetings) — accuracy matters, latency doesn't. Streaming: caption a live conversation as it's spoken, showing words within a fraction of a second — latency is everything. Same model, opposite constraints, and both run heavy GPU inference.
A speech-to-text (ASR) service serves both modes over a GPU model. The shape: a gateway that splits streaming from batch, a batched GPU worker fleet running the model, chunking/VAD to feed it, and post-processing (punctuation, timestamps, diarization) to make raw tokens useful. The hard parts are the streaming latency budget and cost.
How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the pipeline grow, and at the end flood the batch queue and disable post-processing to see path isolation and the value layer. Hit Begin.
Step 1 · Two modes, one model
Batch vs streaming
Before any architecture, recognize that "transcribe audio" is really two problems with opposite priorities. What are they, and why can't one design serve both naively?
Design decision: Why can't a single naive "upload → transcribe → return" design serve all speech-to-text?
The call: Because audio files are too large to upload. — File size is manageable. The real split is latency: real-time captioning can't wait for a whole-file transcription, so it needs incremental, low-latency processing that a simple upload-and-return can't provide.
Two modes, opposite constraints. Batch can wait and use the whole recording for maximum accuracy — an async job. Streaming must emit words as they're spoken, within a fraction of a second, before the sentence is even done — a persistent connection with incremental, low-latency decoding. They share the model but need different plumbing, so the design splits them at the front door.
Latency defines the shape: Batch = throughput/accuracy-optimized async jobs (like the text-to-image queue). Streaming = latency-optimized real-time, emitting partial transcripts that may be revised as more audio arrives. Recognizing these as separate paths (sharing model + fleet) is the key first decision.
Step 2 · The front door
A gateway that splits the paths
Given two modes, what does the entry point do the moment audio (live or uploaded) arrives?
A gateway routes by mode. For streaming, it holds a persistent connection (WebSocket/gRPC) and pipes audio frames toward the workers, returning partial transcripts live. For batch, it accepts the file, enqueues an async job, and returns a job id to poll or a callback URL. Both feed the same GPU model fleet, but through different flow-control: real-time streaming vs buffered async jobs.
One fleet, two front-ends: The gateway is where the two constraints are reconciled: it exposes a low-latency streaming interface and a throughput-oriented batch interface, then multiplexes both onto a shared, expensive GPU fleet. Keeping the interfaces separate lets each meet its own SLA.
Step 3 · Buffer the batch
Async jobs for uploads
Uploaded recordings are heavy (an hour of audio is minutes of GPU work) and arrive in unpredictable bursts. How do you handle them without dropping jobs or starving the latency-sensitive streaming path?
Send batch uploads through a job queue to the GPU workers, exactly like any heavy async job: enqueue, transcribe at the fleet's sustainable rate, deliver via poll or callback. Crucially, keep batch and streaming on separate paths/queues (or reserved capacity) so a flood of uploaded files can't consume the GPUs that real-time captioning depends on. Batch absorbs bursts as backlog; streaming keeps its low-latency lane.
Isolate the SLAs: Batch and streaming compete for the same scarce GPUs but have incompatible SLAs. A shared undifferentiated pool lets a batch flood ruin real-time latency. Separate queues (or capacity reservations / priority) protect the interactive path — the same isolation principle as any mixed-priority workload.
Step 4 · Inside the model
The ASR inference pipeline
A worker has audio to transcribe — what does the model actually do to turn a waveform into words?
Convert the raw waveform into audio features — typically a mel spectrogram (a time-frequency representation) — then feed it to the ASR model (a Whisper-style encoder-decoder or transducer), which produces a sequence of text tokens, decoded (often with beam search and an optional language model) into words. It's a full neural-network inference per segment of audio, which is why it's GPU-bound. Modern models also do language ID and are robust to accents/noise as part of the same pass.
Waveform → features → tokens: ASR is sequence-to-sequence: audio in, text out. The mel spectrogram is the standard input representation; the model attends over it to emit tokens. Compute scales with audio length and model size, so how you segment the audio (next step) directly affects both latency and cost.
Step 5 · Feeding the model
Chunking & voice activity detection
You can't feed an open-ended live stream (or a one-hour file) to the model in one shot — models have a bounded input window, and streaming needs to emit results before the audio ends. How do you segment the audio well?
Design decision: You can't feed unbounded/live audio to a fixed-window model in one shot. How do you segment it?
The call: Use voice-activity detection to split at pauses, with overlapping windows, and decode chunks incrementally. — VAD finds speech vs silence so you can segment at natural pauses (not mid-word), overlap windows so context spans boundaries, and — for streaming — decode each chunk as it arrives, emitting partial transcripts that may be refined as more audio comes in. This balances the model's bounded window against accuracy and latency.
Use voice-activity detection (VAD) to split audio into speech segments at natural pauses (not mid-word), with overlapping windows so context carries across boundaries. For streaming, decode each chunk incrementally as it arrives and emit a partial transcript that can be revised as more audio comes in; finalize when the segment ends. Chunk size is the core dial: smaller = lower latency but less context (lower accuracy), larger = the reverse.
VAD + overlap + partials: Segmentation is where streaming ASR lives or dies. VAD avoids cutting words; overlap preserves cross-boundary context; incremental decoding with revisable partials gives the "words appear as you speak, then settle" experience. The chunk-size knob directly trades latency against accuracy.
Step 6 · Feed the fleet
Batched, autoscaling GPU workers
ASR inference is GPU-heavy, demand fluctuates, and you're paying for expensive accelerators. How do you run the worker fleet economically — while respecting streaming's latency?
Design decision: The same GPU fleet serves both batch and streaming. Should both paths use the same batch size for inference?
The call: No — large batches for the batch path, small/prompt-flushed batches for streaming. — Batching raises throughput but adds latency by design (waiting to fill the batch). Batch jobs are latency-tolerant so they can use large batches for cost efficiency; streaming needs tiny, immediately-flushed batches to keep the sub-second budget, accepting lower GPU efficiency as the price.
Batch multiple requests per GPU to raise utilization (with care: streaming needs small, fast batches to stay low-latency, while batch jobs can use large batches for throughput). Autoscale the fleet on load (queue depth for batch, active-stream count for real-time). Keep the multi-GB model resident (warm pool, avoid cold starts), and consider smaller/faster models or quantization for the streaming path where latency dominates and a slight accuracy trade is acceptable.
Batch for throughput, small for latency: Same GPU-economics as any inference service, with a twist: batching helps throughput but hurts latency, so you use big batches for the batch path and tiny, prompt-flushed batches for streaming. Autoscaling and warm weights keep costs down without breaking the real-time SLA.
Step 7 · Make it useful
Post-processing
The model outputs raw tokens — often lower-cased, unpunctuated, with no idea who spoke or when. That's a wall of text, not a usable transcript. What turns tokens into a product?
Add a post-processing layer: punctuation & capitalization restoration, word-level timestamps (for captions/search), speaker diarization (who spoke which segment), language identification, and optional custom vocabulary / boosting for domain terms and names. This runs after (or alongside) decoding and is what makes a transcript readable, searchable, and attributable. It's a distinct value layer that can degrade gracefully if it fails.
Raw tokens → a transcript: ASR gives you words; the product is punctuated, time-aligned, speaker-attributed text. Diarization ("who spoke") and timestamps are separate models/steps composed on top of recognition. Custom vocab fixes the domain-specific names and jargon that generic models mishear.
Step 8 · The sharp edges
Latency, noise & cost
Real ASR fights hard problems: the streaming latency budget, noisy/overlapping/accented audio, long-form recordings that exceed the model window, and GPU cost.
Hold the streaming latency budget with small chunks, fast/quantized models, and partial-then-refine output — accepting a small accuracy trade for responsiveness. Improve noisy audio with denoising/robust models and VAD, and handle overlapping speakers with diarization. Transcribe long-form audio by sliding the model window with overlap and stitching results. Control cost with batching, right-sized models per path, and caching/dedup of repeated audio. Fail gracefully — degrade post-processing before dropping the core transcript.
Design for the unhappy path: Real-time budget → small chunks + fast models + revisable partials. Noise/overlap → robust models, denoise, diarize. Long audio → sliding window + stitch. Cost → batch, right-size, cache. Streaming ASR is a constant negotiation between latency and accuracy; the knobs above are how you tune it per product.
You did it
You just designed a speech-to-text service.
- Batch (accuracy, latency-tolerant) and streaming (real-time, low-latency) are opposite constraints — split them.
- A gateway routes streaming to a persistent connection and batch to an async job queue, sharing one GPU fleet.
- Keep batch and streaming on isolated paths so a batch flood can't starve real-time latency.
- The model turns a mel spectrogram into text tokens — GPU-bound seq2seq inference.
- VAD + overlapping chunks + incremental partials feed the fixed model window and trade latency vs accuracy.
- Batch on GPUs for throughput but keep streaming batches small; autoscale and keep weights warm.
- Post-processing (punctuation, timestamps, diarization, custom vocab) turns raw tokens into a usable transcript.