Talk is
a stopwatch.

A real-time voice agent is a pipeline. The user speaks; speech-to-text (STT) turns the audio into words; the LLM reads those words and generates a reply; text-to-speech (TTS) turns the reply back into audio the user hears. Three stages, run in order, because each needs the previous one's output: the LLM can't think until it has a transcript, TTS can't speak until it has text.

The consequence is simple and unforgiving: the user hears nothing until all three have finished, so the response time they feel is the sum of the stage times. It doesn't matter that your LLM is brilliant if the round trip takes two seconds — by then the human has already felt the silence. That's what makes voice different from a chat interface, where a spinner is socially acceptable. In conversation, the clock is running the whole time, and the budget for that clock is tiny.

Human turn-taking is astonishingly fast. In natural dialogue people begin replying within a couple hundred milliseconds of the other person stopping — often before, by predicting the end of the sentence. A gap much beyond that starts to feel like something's wrong: the other person is confused, distracted, or the line is bad. Cross about a second and the illusion of conversation collapses entirely.

So a voice agent has a latency budget — a hard ceiling on the end-to-end response time, commonly targeted around 800 ms to feel snappy (and every millisecond under that feels better). Every stage spends against it: STT might take 150 ms, the LLM 400 ms, TTS 200 ms. Add them up and you're at 750 ms — under budget, barely. Swap in a slower LLM at 700 ms and you're at 1050 ms, over the line, and the agent now feels like a laggy phone tree no matter how good its answers are. The whole engineering problem is keeping that sum under the ceiling.

When the sum is over budget, not all stages are equal. The one to fix first is the bottleneck — the slowest stage — because a fixed reduction there moves the total the same as anywhere else, but the bottleneck is where the biggest reductions are available. If the LLM is 700 ms and STT/TTS are 150 ms each, shaving the LLM is the only move that can bring a 1000 ms pipeline under 800 ms; trimming an already-fast stage barely helps.

So the optimization loop is: measure each stage, find the max, cut it, remeasure. Sometimes the bottleneck is the LLM (use a faster model, or fewer tokens); sometimes it's a network hop (colocate the services); sometimes it's TTS warmup. The discipline is to always spend your effort on the slowest stage, because that's the only place the total budget has real slack to give.

The stages run in series, so end-to-end latency is their sum, and the pipeline "fits" only when that sum is within the conversational threshold:

150 + 400 + 200 = 750 ms ≤ 800 → fits. Swap the LLM to 700 → 1050 > 800 → over budget, feels laggy.

The stage to optimize is the one that dominates the sum — the max — and shaving the bottleneck is what pulls an over-budget pipeline back under the line:

Bottleneck of (150, 700, 200) is the LLM; halving it to 350 turns a failing 1050 ms into a passing 700 ms. The runnable version below computes the total, the fit, the bottleneck, and the headroom.

The math above treats the stages as strictly sequential — a worst case. Production voice agents beat it with streaming: each stage emits partial output as it works instead of waiting to finish, so the stages overlap. Streaming STT sends transcript fragments while the user is still talking, so the LLM can start on the first words before the sentence ends. A streaming LLM emits tokens one at a time, and streaming TTS speaks those tokens as they arrive — so the user starts hearing a reply while the LLM is still generating the rest of it.

The effect is dramatic: the perceived time-to-first-audio becomes far smaller than the naive sum, because the user only waits for the pipeline to begin producing sound, not to fully complete. This is the main reason a voice agent can feel real-time even when each stage individually takes a few hundred milliseconds. The sequential sum is still the right mental model for the budget — it's the ceiling you'd hit without overlap — but streaming is the lever that lets a real system live comfortably under it.

The first trap is optimizing the wrong stage. It's tempting to tune the part you understand best, but if you shave 50 ms off a 150 ms STT while a 700 ms LLM sits untouched, you've barely moved the total. Always measure all three and attack the max. The second is ignoring the network: STT, LLM, and TTS are often three separate services, and every hop between them adds real time. Colocating them, or cutting round trips, can matter as much as the model choice.

Two more that make voice feel broken even under budget. Turn-taking: the agent must know when the user has actually stopped speaking (voice-activity detection and endpointing) and must gracefully handle being interrupted — a human who keeps talking over you feels awful, and so does an agent that won't yield. And no thinking-sound: because the budget is so tight, good voice agents fill unavoidable gaps with a natural filler ("let me check…") so the silence never reads as a freeze. Get the budget right and the interruption handling humane, and a voice agent stops feeling like a phone menu and starts feeling like a conversation. The rule is the whole handbook: keep the sum under the clock, and always cut the slowest stage first.