The big idea

What makes a voice agent hard?

Talking to an AI out loud has a brutal constraint humans enforce unconsciously: if the reply doesn't start within roughly half a second to a second of you finishing, the silence feels broken — and if the agent talks over you, or won't stop when you interrupt, it feels worse than useless. Under the hood, that means running speech-to-text → LLM → text-to-speech, end to end, fast enough to feel like a conversation.

A realtime voice agent chains three heavy models into a sub-second loop. The winning moves: stream and pipeline all three stages (never run them sequentially), nail turn-taking (endpointing + barge-in), and coordinate it all with an interruptible orchestrator. Latency and turn-taking — not the models themselves — are the real design.

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the loop grow, and at the end break turn-taking and disable streaming TTS to see what makes it feel human vs robotic. Hit Begin.

Step 1 · Why the obvious way fails

The sequential pipeline is too slow

The naive design: wait for the user to finish, transcribe the whole utterance, send it to the LLM, wait for the full reply, synthesize the whole reply, then play it. Every box is correct. Why does the conversation feel dead?

Design decision: Wait-for-full-utterance → STT → LLM → TTS → play, each stage fully before the next. Why does it feel robotic?

The call: The models are too inaccurate. — Accuracy isn't the issue here; latency is. Even with perfect models, doing STT then LLM then TTS strictly in sequence produces seconds of dead air per turn.

Run strictly in sequence and the latencies stack: wait for the whole utterance + full transcription + full LLM reply + full synthesis before a single sound plays. Even with fast models, the sum is seconds of silence per turn — far past the conversational budget. The fix isn't faster boxes; it's overlapping them so work in one stage begins before the previous finishes.

Sequential = sum of latencies: Three multi-hundred-millisecond stages in series is a multi-second turn. The only way under the budget is to pipeline: stream partial output from each stage into the next so they run concurrently, not one-after-another. Streaming is the whole game.

Step 2 · The loop and its transport

A brain over a live audio link

Before optimizing latency, set the frame: audio must flow both ways in real time, and something must coordinate the three models and the conversation state. What are those two pieces?

A transport carries live audio both directions with minimal latency — WebRTC for the web, SIP for the phone network — streaming the mic up and playing synthesized audio down. And an orchestrator (dialog manager) sits behind it: it drives the STT→LLM→TTS loop, holds conversation state/context, and — critically — keeps the whole thing interruptible. The transport moves sound; the orchestrator runs the conversation.

Transport + orchestrator: Real-time audio needs a real-time transport (UDP-based WebRTC/SIP, like a call), not request/response HTTP. The orchestrator is the stateful conductor that turns three stateless model calls into one coherent, interruptible conversation. Everything else hangs off these two.

Step 3 · Overlap everything

Stream and pipeline the three stages

You've decided sequential is too slow. Concretely, how do you overlap STT, the LLM, and TTS so the agent starts replying almost as the user stops talking?

Design decision: How do you overlap STT, LLM and TTS to cut per-turn latency?

The call: Stream each stage: STT emits partial text → LLM streams tokens as it reads → TTS synthesizes+plays the first chunk immediately, all overlapping. — STT emits partial transcripts as the user speaks, so the LLM can begin as soon as the utterance is (semantically) complete; the LLM streams tokens; and TTS turns the first tokens into audio and starts playing while the LLM is still generating. The stages run concurrently, so end-to-end latency approaches just the first chunk of each — not their sum.

Stream every stage and pipeline them. STT emits partial transcripts live; the moment the user's turn ends, the LLM starts and streams tokens; those tokens flow into TTS, which synthesizes and plays the first audio chunk immediately while the LLM is still generating the rest. The stages run concurrently, so time-to-first-audio approaches the first chunk of each stage rather than the sum of all three. That's how the agent replies almost instantly.

Pipeline = concurrency: Streaming lets stage N+1 start on stage N's partial output. STT partials warm up the LLM; LLM tokens feed TTS; TTS plays while the LLM thinks. The per-turn latency becomes roughly the slowest single first-chunk plus small overlaps — not STT + LLM + TTS end-to-end.

Step 4 · Knowing when to talk

Turn-taking: endpointing & barge-in

Humans hand off turns effortlessly; machines don't. The agent must know when the user has finished speaking (so it replies at the right moment, not mid-sentence or after an awkward pause) and must stop instantly when the user interrupts. How?

Design decision: How does the agent know when to start replying, and stop when interrupted?

The call: Reply after a fixed 2-second pause every time. — A fixed timeout is either too slow (dead air after short answers) or cuts people off mid-thought, and it ignores interruptions entirely. Turn-taking must be adaptive (VAD + semantic endpointing) and support barge-in.

Solve turn-taking explicitly. Endpointing decides the user's turn is over — voice-activity detection catches the silence, refined by whether the utterance is semantically complete — so the agent replies promptly without cutting the user off. Barge-in detects the user speaking while the agent is talking and immediately stops TTS, flushing the pending reply and re-listening. This interruptibility is what makes the agent feel like a person, not a recording.

Endpoint + barge-in: Two hard human behaviors made explicit: when did you finish? (endpointing — VAD plus a semantic check so a mid-sentence pause isn't mistaken for the end) and you jumped in, I'll stop (barge-in — halt playback and generation on detected user speech). Getting these right matters more to "feeling human" than raw model quality.

Step 5 · The latency budget

Where the milliseconds go

"Under a second" is a budget you have to spend across STT, LLM first token, and TTS first audio — plus network. If any stage blows its slice, the whole turn feels laggy. How do you make it fit?

Break the end-to-end budget into slices and optimize each: STT partial/final latency, LLM time-to-first-token (the usual biggest chunk — use a fast model, short prompts, prefix/KV caching, low-latency serving), TTS time-to-first-audio, plus network/transport. Because you pipeline, the turn latency is dominated by the LLM's first token plus the first TTS chunk — so you attack those hardest. Measure the whole loop, not each box, and keep a tight budget (often targeting a few hundred ms of perceived response start).

Optimize the first chunk, not the total: With pipelining, what the user feels is time-to-first-audio: how fast the agent starts speaking. That's roughly LLM first-token + TTS first-chunk (STT overlaps with the user still finishing). So you spend your optimization budget on LLM first-token latency and TTS first-chunk — the two that gate the start of the reply.

Step 6 · Interruptible by design

The orchestrator holds it together

Streaming three models concurrently, mid-turn interruptions, tool calls, and conversation state make this genuinely stateful and racy. What does the orchestrator actually manage?

The orchestrator is the stateful conductor. It manages the conversation context (history, system prompt, user/session memory), sequences the streaming pipeline, and enforces interruptibility: on barge-in it must cancel in-flight LLM generation and TTS playback cleanly, keep only what was actually spoken as context, and re-listen. It handles turn state (listening / thinking / speaking), coordinates tool calls (pausing the voice while a function runs, or speaking a filler), and recovers from stage errors — all without dropping the thread of the conversation.

State + cancellation: Real-time + interruptible = careful state management and clean cancellation. When the user barges in, half-generated tokens and queued audio must be discarded, and the context updated to what was truly heard/said. The orchestrator owns this; get cancellation wrong and the agent "remembers" things it never actually said.

Step 7 · Doing things, not just talking

Tools, integration & transport

A useful voice agent doesn't just chat — it looks up your order, books the appointment, checks the balance. And it lives on real channels (a phone line, a web widget). How does action and integration fit into a latency-sensitive loop?

Give the LLM tools (function calling) to query backends and take actions mid-conversation. Because a tool call adds latency, the orchestrator masks it: speak a natural filler ("let me check that…") or acknowledge while the function runs, then continue with the result — so the pause feels intentional. Integrate with telephony (SIP) for phone agents and WebRTC for web, and connect to knowledge (RAG) and memory for continuity. The agent becomes a spoken interface to real systems, not just a talking LLM.

Tools + graceful latency: Function calling turns the agent into an actor. The trick is hiding tool latency behind natural conversational filler so the user never hears dead air. Transport (SIP/WebRTC), RAG for knowledge, and memory for continuity make it a real product on real channels.

Step 8 · The sharp edges

Noise, overlap, cost & end-to-end models

Live conversation is messy: background noise and cross-talk confuse endpointing, interruptions race with generation, three model calls per turn cost money and can each fail, and a new class of models threatens to collapse the whole pipeline.

Handle noise/cross-talk with robust VAD, echo cancellation, and denoising so the agent doesn't mistake noise for speech (or barge-in). Make interruptions race-safe with clean cancellation. Control cost (three inferences per turn) with right-sized models, caching, and ending calls gracefully. Build in error recovery — if a stage fails, apologize/retry rather than freeze. And watch the frontier: end-to-end speech-to-speech models (audio in, audio out, no separate STT/LLM/TTS) can cut latency and preserve tone/emotion — a possible replacement for the pipeline, with less modular control.

Design for the unhappy path: Noise → robust VAD + denoise + echo cancel. Interrupts → race-safe cancellation. Cost → right-size the three models. Failure → recover, don't freeze. And the pipeline itself may give way to unified speech-to-speech models that are lower-latency and more expressive but harder to steer. The architecture here is the current best trade, not the final word.

You did it

You just designed a realtime voice agent.

• A —
• A —
• S — t
• T — u
• S — p
• T — h
• T — o