The whole design, in writing
Learn AI system design by building a realtime voice AI agent like ChatGPT voice mode or an AI phone agent step by step. An interactive guide covering why a sequential pipeline feels robotic, streaming and pipelining STT→LLM→TTS, turn-taking and barge-in, the dialog orchestrator, the end-to-end latency budget, tool calling and transport (WebRTC/SIP), and the unhappy paths (interruptions, noise, cost, end-to-end speech models).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What makes a voice agent hard?
Talking to an AI out loud has a brutal constraint humans enforce unconsciously: if the reply doesn't start within roughly half a second to a second of you finishing, the silence feels broken — and if the agent talks over you, or won't stop when you interrupt, it feels worse than useless. Under the hood, that means running speech-to-text → LLM → text-to-speech, end to end, fast enough to feel like a conversation.
A realtime voice agent chains three heavy models into a sub-second loop. The winning moves: stream and pipeline all three stages (never run them sequentially), nail turn-taking (endpointing + barge-in), and coordinate it all with an interruptible orchestrator. Latency and turn-taking — not the models themselves — are the real design.
What the new pieces do
- Userclient
- A person having a spoken conversation with the agent — over the phone or the web. Expects human-like, low-latency back-and-forth.
Step 1 · Why the obvious way fails
The sequential pipeline is too slow
The naive design: wait for the user to finish, transcribe the whole utterance, send it to the LLM, wait for the full reply, synthesize the whole reply, then play it. Every box is correct. Why does the conversation feel dead?
Wait-for-full-utterance → STT → LLM → TTS → play, each stage fully before the next. Why does it feel robotic?
Running each stage to completion before starting the next stacks their latencies, so the user hears a long gap after every utterance. Even if each model is fast, the sum blows the conversational budget. You must overlap the stages by streaming.
Accuracy isn't the issue here; latency is. Even with perfect models, doing STT then LLM then TTS strictly in sequence produces seconds of dead air per turn.
Quality doesn't cause the awkward pauses. The problem is that sequential, non-streaming stages sum their latencies into an unnatural silence after every turn.
Run strictly in sequence and the latencies stack: wait for the whole utterance + full transcription + full LLM reply + full synthesis before a single sound plays. Even with fast models, the sum is seconds of silence per turn — far past the conversational budget. The fix isn't faster boxes; it's overlapping them so work in one stage begins before the previous finishes.
Why this piece earns its place
The sequential build is worth doing once, because it is the one you can debug, and it is worth knowing when to keep. Batch speech-to-text reading a whole utterance is more accurate than the partials a streaming decoder emits, and for voicemail transcription, dictation or call summarization nobody is waiting on the other end — there the simple pipeline is the right answer and overlapping is wasted complexity. What makes it a trap for realtime is that nothing about it looks broken: three green stage dashboards and one unusable product. The heavier cost is structural, and it is the one that decides a quarter rather than an afternoon. Once the seams are shaped as transcribe, then generate, then speak, each handing on a finished value, streaming is not a flag you flip — it is a rewrite of every interface between the stages, and of the error handling with them, because a stage whose output has already reached a human cannot be quietly retried. Shape the seams as streams while the pipeline is still small, or pay for it later as a migration.
Step 2 · The loop and its transport
A brain over a live audio link
Before optimizing latency, set the frame: audio must flow both ways in real time, and something must coordinate the three models and the conversation state. What are those two pieces?
A transport carries live audio both directions with minimal latency — WebRTC for the web, SIP for the phone network — streaming the mic up and playing synthesized audio down. And an orchestrator (dialog manager) sits behind it: it drives the STT→LLM→TTS loop, holds conversation state/context, and — critically — keeps the whole thing interruptible. The transport moves sound; the orchestrator runs the conversation.
Why this piece earns its place
Start with what the models actually hear, because the transport decides it and nothing downstream gets a vote. A browser hands you wideband audio; a phone leg might too, over HD voice, or might hand you a narrowband codec that has already thrown away the top of the spectrum — and the same agent degrades measurably on the narrow leg for reasons that have nothing to do with the speech-to-text model you picked. If you promise SIP, test on SIP. Operationally this tier is stranger than it looks, because a call is a pinned session and not a request. Audio flows for the whole conversation into one process holding one piece of conversation state, so you size on concurrent calls rather than requests per second, and a rollout has to drain calls instead of cycling instances — there is no retry that puts a dropped conversation back. The dial to know by name here is the jitter buffer: it trades smoothness against delay, it is sized against the network rather than against your turn budget, and a bad connection quietly buys audio quality with milliseconds you had earmarked for the models.
- WebRTC / SIPlive audio both ways
- orchestratorruns the loop + state
- interruptiblestop on barge-in
What the new pieces do
- Transportgateway
- Carries live audio both ways with low latency (WebRTC for web, SIP for phone). Streams mic in, plays synthesized audio out.
- Orchestratorservice
- The brain: pipelines STT→LLM→TTS, holds conversation state, and enforces interruptibility — stop everything when the user barges in.
Step 3 · Overlap everything
Stream and pipeline the three stages
You've decided sequential is too slow. Concretely, how do you overlap STT, the LLM, and TTS so the agent starts replying almost as the user stops talking?
How do you overlap STT, LLM and TTS to cut per-turn latency?
Smaller models help a little but the stages still run in series, still summing latencies. The structural fix is to stream partial outputs so stages overlap — pipelining, not just speed.
Co-locating doesn't change that output of one stage feeds the next; without streaming they still wait for each other. You need incremental hand-off between stages.
STT emits partial transcripts as the user speaks, so the LLM can begin as soon as the utterance is (semantically) complete; the LLM streams tokens; and TTS turns the first tokens into audio and starts playing while the LLM is still generating. The stages run concurrently, so end-to-end latency approaches just the first chunk of each — not their sum.
Stream every stage and pipeline them. STT emits partial transcripts live; the moment the user's turn ends, the LLM starts and streams tokens; those tokens flow into TTS, which synthesizes and plays the first audio chunk immediately while the LLM is still generating the rest. The stages run concurrently, so time-to-first-audio approaches the first chunk of each stage rather than the sum of all three. That's how the agent replies almost instantly.
Why this piece earns its place
What separates this from a demo is what you let the system do with output that is not final yet. Partial transcripts are provisional — a streaming recognizer revises text it already emitted as more audio arrives, so anything the LLM starts on a partial may have been started on the wrong sentence. Speculative starts are real and useful, and the line to hold is side effects: a speculation you can discard costs nothing but work, while a speculative turn that reaches a tool call has already changed something in the world and cannot be un-run. Speculate up to the first thing that touches a backend, and stop there. Then there is the audio queue. Synthesis runs faster than real time, so the further ahead TTS works, the more finished audio sits waiting to be played. Deep buffering makes playback smooth and makes an interruption expensive, because everything queued has to be discarded and accounted for. Shallow buffering makes barge-in cheap and exposes every synthesis hiccup as a gap mid-sentence. That one number is the clearest case on this page of a choice made here and paid for three steps later.
- STT partialsfeed LLM early
- LLM tokensstream to TTS
- first audioplays while generating
What the new pieces do
- Streaming STTservice
- Transcribes the user's speech incrementally, emitting partial text so the LLM can start thinking before the sentence ends.
- Streaming TTSservice
- Synthesizes the reply in chunks and streams the first audio out immediately, so the agent starts speaking fast.
- Streaming LLMservice
- Generates the reply token-by-token; its streamed tokens feed TTS as they arrive, overlapping thinking with speaking.
Step 4 · Knowing when to talk
Turn-taking: endpointing & barge-in
Humans hand off turns effortlessly; machines don't. The agent must know when the user has finished speaking (so it replies at the right moment, not mid-sentence or after an awkward pause) and must stop instantly when the user interrupts. How?
How does the agent know when to start replying, and stop when interrupted?
Endpointing decides the user's turn is over — silence via VAD, refined by whether the utterance is semantically complete — so the agent replies at the right moment. Barge-in detects the user talking while the agent speaks and immediately halts TTS and re-listens. Together they create natural turn-taking.
A fixed timeout is either too slow (dead air after short answers) or cuts people off mid-thought, and it ignores interruptions entirely. Turn-taking must be adaptive (VAD + semantic endpointing) and support barge-in.
Push-to-talk defeats the point of a natural voice conversation. A good agent handles turn-taking automatically, including being interrupted.
Solve turn-taking explicitly. Endpointing decides the user's turn is over — voice-activity detection catches the silence, refined by whether the utterance is semantically complete — so the agent replies promptly without cutting the user off. Barge-in detects the user speaking while the agent is talking and immediately stops TTS, flushing the pending reply and re-listening. This interruptibility is what makes the agent feel like a person, not a recording.
Why this piece earns its place
The part that bites is that not all user speech is an interruption. People say mhm, right, yeah while they are listening — backchannels, the noises that mean keep going — and a detector that fires on voice activity alone stops the agent dead every time the user agrees with it. You want at least a duration floor, and better, a look at what was actually said, before tearing a turn down. The two ways endpointing gets it wrong are also not symmetric, which is worth saying out loud. Waiting a beat too long costs a beat. Cutting in early costs the rest of the sentence, makes the user say it again, and leaves the orchestrator holding half an utterance that it will answer as though it were whole. So bias the threshold toward waiting, then buy the wait back per turn — because the agent knows what it just asked. After a request for an account number, people pause between digit groups and you should be patient; after a yes-or-no question, a short silence is already the entire answer. Make the wait a value the dialog sets on the turn rather than a constant in the config, and a good share of perceived lag disappears without touching a model.
- endpointingVAD + semantics
- barge-instop TTS on interrupt
- re-listenflush pending reply
What the new pieces do
- Turn-takingservice
- Detects when the user stops speaking (endpointing) and when they interrupt (barge-in), so the agent knows when to speak and when to stop.
Step 5 · The latency budget
Where the milliseconds go
"Under a second" is a budget you have to spend across STT, LLM first token, and TTS first audio — plus network. If any stage blows its slice, the whole turn feels laggy. How do you make it fit?
Break the end-to-end budget into slices and optimize each: STT partial/final latency, LLM time-to-first-token (the usual biggest chunk — use a fast model, short prompts, prefix/KV caching, low-latency serving), TTS time-to-first-audio, plus network/transport. Because you pipeline, the turn latency is dominated by the LLM's first token plus the first TTS chunk — so you attack those hardest. Measure the whole loop, not each box, and keep a tight budget (often targeting a few hundred ms of perceived response start).
Why this piece earns its place
First token is the right shorthand above and not quite the mechanism. What gates audio is the first speakable unit: handed a single token, a synthesizer has no phrase to place stress or intonation on, so low-latency voices buffer to a clause or run a little lookahead, trading a slice of prosody for every millisecond they shave. Treat the gate as the opening clause and the shape of the first sentence becomes a latency decision — a reply that starts with a short acknowledgement makes sound sooner than one that starts with a long qualifying clause, at no cost to the answer. That is the cheapest millisecond on this page. The other thing to hold onto is that a budget written as one number is a median, and nobody experiences a conversation at the median. Three model calls per turn means the turn's tail is the worst of three tails rather than the average of them, so the spread is wider than any single stage suggests, and the one bad turn is the one a user recounts afterwards. Defend the budget at a high percentile, and treat a tail that moves while the median holds as a real regression.
- budget~500ms–1s end-to-end
- LLM TTFTusually the biggest slice
- measurethe whole loop
Step 6 · Interruptible by design
The orchestrator holds it together
Streaming three models concurrently, mid-turn interruptions, tool calls, and conversation state make this genuinely stateful and racy. What does the orchestrator actually manage?
The orchestrator is the stateful conductor. It manages the conversation context (history, system prompt, user/session memory), sequences the streaming pipeline, and enforces interruptibility: on barge-in it must cancel in-flight LLM generation and TTS playback cleanly, keep only what was actually spoken as context, and re-listen. It handles turn state (listening / thinking / speaking), coordinates tool calls (pausing the voice while a function runs, or speaking a filler), and recovers from stage errors — all without dropping the thread of the conversation.
Why this piece earns its place
Cancellation carries one subtlety that decides whether the transcript is even true. What was spoken is not what the synthesizer emitted — audio sits in the send path and in the listener's playback buffer, so at the instant of a barge-in the user has heard strictly less than the server produced. The cut point therefore has to come back from the playback clock: the client reports how far it actually got, and the orchestrator maps that position back onto the text it queued. That reporting path is real engineering, and it is the part people leave out of the design — without it the orchestrator has only a server-side count of what it generated, which is the wrong number. Interruption is not free either. A cancelled generation still bills the tokens it produced, and truncating the last assistant turn invalidates the cached prefix from that turn onward — the system prompt and every earlier exchange still match and are still reused, so what gets re-prefilled is the shortened tail plus the new user turn. On discarded output alone, a heavily interrupted call costs more per turn than a polite one, and that is the call your unhappiest user is having.
- statelistening/thinking/speaking
- cancelclean barge-in teardown
- contextonly what was really said
Step 7 · Doing things, not just talking
Tools, integration & transport
A useful voice agent doesn't just chat — it looks up your order, books the appointment, checks the balance. And it lives on real channels (a phone line, a web widget). How does action and integration fit into a latency-sensitive loop?
Give the LLM tools (function calling) to query backends and take actions mid-conversation. Because a tool call adds latency, the orchestrator masks it: speak a natural filler ("let me check that…") or acknowledge while the function runs, then continue with the result — so the pause feels intentional. Integrate with telephony (SIP) for phone agents and WebRTC for web, and connect to knowledge (RAG) and memory for continuity. The agent becomes a spoken interface to real systems, not just a talking LLM.
Why this piece earns its place
Voice changes the contract a tool has to meet. There is no screen, so a function that returns ten options or a page of structured data forces the model to compress it under time pressure, and what it drops is whatever it judged least interesting, which is not the same as least important. Shape voice tools to return few fields, already narratable. The sharper issue is that speech has no confirmation dialog and no undo. Recognition mishears exactly the values that matter — fifteen and fifty, B and D, one digit inside an account number — so anything that moves money or changes state should read the parsed arguments back and wait for an explicit yes. That round trip, not the function call, is where the real latency of an action goes, and it is the one spend here worth defending. Last, the filler is a promise. Once the agent has said it is checking, silence stops being neutral, so every tool call needs a deadline and a line to say when it expires.
- toolslook up / act
- fillermask tool latency
- SIP / WebRTCphone + web
What the new pieces do
- Toolsdata
- Function calls the agent can make mid-conversation — look up an order, book a slot, query a backend — grounding the dialog in real actions.
Step 8 · The sharp edges
Noise, overlap, cost & end-to-end models
Live conversation is messy: background noise and cross-talk confuse endpointing, interruptions race with generation, three model calls per turn cost money and can each fail, and a new class of models threatens to collapse the whole pipeline.
Handle noise/cross-talk with robust VAD, echo cancellation, and denoising so the agent doesn't mistake noise for speech (or barge-in). Make interruptions race-safe with clean cancellation. Control cost (three inferences per turn) with right-sized models, caching, and ending calls gracefully. Build in error recovery — if a stage fails, apologize/retry rather than freeze. And watch the frontier: end-to-end speech-to-speech models (audio in, audio out, no separate STT/LLM/TTS) can cut latency and preserve tone/emotion — a possible replacement for the pipeline, with less modular control.
Why this piece earns its place
The trade nobody names when speech-to-speech comes up is observability. In this pipeline the transcript is a by-product you get for free, and almost everything downstream leans on it: quality review, offline evaluation, compliance recording, analytics, and the whole story for debugging a bad turn, where you can point at text and say the transcription was fine, the reply drifted. A model that takes audio in and returns audio out removes that seam, and bolting a transcription pass beside it does not hand the seam back. That pass runs off the conversational path, so what it costs is money and a second system to keep aligned rather than time-to-first-audio — but what it returns is a record of what was said, not of what the model was reasoning over. You can still see the bad turn and still not know which half of it went wrong. One correction to the cost line above, too: right-sizing the three models is a win you collect once, while the bill that keeps growing is context. Each turn re-sends the conversation so far, so a late exchange in a long call costs far more than the first, which makes the orchestrator's summarization policy the lever that keeps paying.
You did it
You just designed a realtime voice agent.
Everything you assembled, in order
- A sequential STT→LLM→TTS pipeline sums latencies into seconds of dead air — it feels robotic.
- A low-latency transport (WebRTC/SIP) + a stateful orchestrator frame the real-time loop.
- Stream and pipeline all three stages so they run concurrently — time-to-first-audio, not the sum.
- Turn-taking is explicit: endpointing (when the user finished) + barge-in (stop instantly on interrupt).
- Spend the latency budget on LLM first-token and TTS first-chunk — the two that gate the reply start.
- The orchestrator manages state and clean cancellation so barge-in never corrupts the conversation.
- Tools (with filler to mask latency) make it act; watch end-to-end speech-to-speech models emerging.