The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
A conversation, not a walkie-talkie
Chain speech-to-text, translation, and text-to-speech together and you have SOMETHING — but "wait for the speaker to finish, then translate the whole thing, then speak it" produces a multi-second lag per exchange that makes a real back-and-forth conversation feel like a walkie-talkie: talk, wait, listen, wait, repeat.
We'll build it to actually stream: partial transcripts, incremental translation that revises itself as more context arrives, careful chunking, voice-preserving synthesis, and an explicit latency budget split across every stage.
What the new pieces do
- Speakerclient
- Someone speaking naturally, in real time, expecting the translation to keep pace with a real conversation, not lag behind it.
Step 1 · The baseline
Record, then translate, then speak
Simplest version: wait for the speaker to finish a full utterance, transcribe it, translate the complete text, synthesize speech, then play it. What's the actual cost of waiting for completeness?
The pipeline waits for a full utterance before doing anything. What's the cost?
Accuracy from having full context is real, but it's bought at the cost of the ENTIRE utterance's length plus processing time before the listener hears anything — for a long sentence, that's multiple seconds of dead air, incompatible with real conversational pacing.
Waiting for completeness means the listener's first sound arrives only after the ENTIRE pipeline has processed the ENTIRE utterance — for anything but very short sentences, that delay breaks the natural overlap and pacing of real conversational turn-taking.
Even ordinary conversational sentences (5-10 seconds of speech) produce a noticeable, conversation-breaking delay under this approach — it's not just a long-speech problem.
Waiting for a complete utterance stacks its full length plus every stage's processing time before the listener hears anything — for real conversational pacing, that delay is disqualifying. The entire pipeline needs to stream, producing partial output as input arrives, not batch-process complete utterances.
Why this piece earns its place
Streaming is not a faster version of the batch pipeline — it changes what every stage is. A batch stage is a request: complete input in, complete output out, retryable, load-balanced per call, drained cleanly on deploy. A streaming stage is a long-lived session holding audio state, so the connection is sticky, an autoscaler cannot move it mid-utterance, and a rolling restart cuts people off mid-sentence unless you learn to drain conversations rather than instances. Retries get worse too: a failed batch call re-runs invisibly, but you cannot retry a chunk the listener already heard, so recovery becomes a correction rather than a repeat. Capacity accounting changes shape — a slot is held for as long as the session is open, not for the length of one call to a model, so utilization tracks connected time rather than request rate. Silence costs exactly what speech costs, which means a sparse meeting and a dense one occupy identical capacity while producing wildly different amounts of translated speech, and the number you provision against is concurrent conversations, not traffic. The one case to skip all of this: nobody is waiting. Dubbing recorded video or translating voicemail has no live listener, and there batch buys better quality, simpler operations and cheaper compute. Paying for streaming with no one on the other end is pure complexity.
What the new pieces do
- Streaming ASRgateway
- Converts speech to text incrementally, emitting partial transcripts as words arrive, not waiting for the speaker to finish.
- Incremental MTservice
- Translates partial transcripts as they arrive — necessarily working with incomplete information, which it will need to revise.
- Streaming TTSservice
- Synthesizes the translated text into speech, ideally preserving something of the original speaker's voice and delivery.
- Listenerclient
- Hears the translated speech and needs it to feel close enough to real time to sustain an actual conversation.
Step 2 · Streaming ASR
Partial transcripts, not one final answer
Fix the biggest lag source first: don't wait for the speaker to finish before transcribing.
Should transcription wait for the speaker to pause or finish, or produce output continuously?
Waiting for pauses still reintroduces meaningful per-segment latency, and natural conversational speech often has few clean pauses within a longer thought — you'd still be waiting a long time in many real utterances.
Streaming ASR treats transcription as a continuously-updating process rather than a single final answer — early partial text may be revised as more context arrives, but something is always available to feed downstream immediately, which is what unlocks the rest of the pipeline streaming too.
Fixed time-based chunking with no regard for actual speech boundaries can cut words or short phrases mid-way, producing transcription errors that have nothing to do with the audio quality — chunking needs to be informed by the audio, not arbitrary.
Make ASR streaming: it continuously emits partial transcripts as audio arrives, revising earlier words if later context changes the most likely interpretation. This gives downstream translation something to work with immediately, rather than only after the speaker stops.
Why this piece earns its place
What makes partials usable downstream is a stability marker — ASR saying which prefix it considers committed and which it may still rewrite. Without one, every consumer guesses. The translator re-translates text that was never going to change, burning compute on identical input, and the renderer has to treat the whole line as provisional, repainting settled words alongside the tail that is genuinely still moving. That indiscriminate repaint is the flicker, and it is the failure nobody tracks: word error rate is measured on the final transcript and looks excellent while the partials thrash, because the listener watched every draft and the metric never saw one of them. With the marker, churn is scoped to the words still in play, which is what lets the next stage hold captions and audio to different release rules at all. Two operational notes worth having ready. Streaming decoding re-runs the model over overlapping audio, so compute per second of speech is a multiple of the batch cost for the same words — that lands as GPU spend, not as latency, which is why it survives a latency review. And partials usually arrive without punctuation or casing, since restoring those is a separate model, which matters here because the next stage segments on exactly those cues. Whether that model runs on partials or only on committed text is the choice that sets how much of the pipeline is allowed to churn.
Step 3 · Incremental translation must revise itself
Word order isn't the same across languages
Translating a partial sentence before it's complete forces the translator to guess. In German, the main verb often comes at the END of a clause — translating the first few words into English before seeing that verb can require a fundamentally different sentence structure than what was guessed. What should happen?
A partial translation was based on an early guess that later context contradicts. What should happen?
Locking in an early, possibly-wrong guess (especially for languages with meaningfully different word order) guarantees the FINAL translation is sometimes simply incorrect — accuracy has to matter more than never revising, or the whole point of translation is undermined.
Because early-arriving words genuinely may not determine the correct translation until more context (like a German sentence-final verb) arrives, the system needs to treat its own partial output as provisional and revise it — the alternative is either high latency (wait for certainty) or wrong output (never revise).
This just reverts to Step 1's full-utterance-latency problem — the whole point of streaming translation is to start producing SOME output before the sentence is complete, while accepting that output may need revision.
Add a Re-translation Stabilizer: as more of the sentence arrives, it re-evaluates whether the current translation is still the best one and revises the displayed or spoken output if a meaningfully better translation is now possible — accepting a degree of self-correction as the necessary cost of low latency, rather than either waiting for certainty or locking in an early wrong guess.
Why this piece earns its place
Revision is cheap in pixels and impossible in audio. A caption can be rewritten in place; a spoken sentence has already left the room and cannot be un-said. So the Stabilizer does not have one output policy, it has two — captions take revisions as they land, repainting only the unstable tail that Step 2's marker already scopes for them, while the audio path is handed nothing until the Stabilizer has committed it. That single decision splits the system into two clocks: the caption track runs ahead, the voice track runs behind by however long the commit rule withholds text, and you have to say which one the latency budget is measured against or the number means nothing. Two things fall out of it. Every emitted segment needs a revision id so the renderer replaces rather than appends — a naive append turns each correction into visible duplicate text. And the rewrite window has to be bounded: how far back may a late word reach? Unbounded means the Stabilizer holds the whole utterance and its work grows with sentence length. Bounded means some genuinely correct late fixes are dropped on purpose, and that is the honest trade.
What the new pieces do
- Re-translation Stabilizerservice
- Re-translates a segment as more of the sentence arrives, since word order across languages can make an early partial translation wrong.
Step 4 · Chunking policy
Not word-by-word, not full-sentence
How large should each chunk of text handed to translation be — every single word the instant it's transcribed, or wait for a reasonably complete phrase?
What's the right granularity for feeding transcribed text into the translator?
Word-by-word translation, with essentially zero context per word, produces extremely poor and constantly-revising output — most words' correct translation genuinely depends on surrounding words, which single-word chunks don't provide.
Phrase-level chunking gives the translator enough local context to produce a reasonably confident translation while still emitting output well before the full sentence completes — a deliberate middle ground between the two extremes, tuned to the actual latency/quality trade-off.
This reverts to Step 1's core latency problem — sentence-level chunking defeats the purpose of streaming translation entirely.
Chunk at natural phrase or clause boundaries — enough accumulated context for a reasonably confident translation, without waiting for full-sentence completeness. This is a tunable policy sitting deliberately between word-by-word (too little context, too much revision) and full-sentence (too much latency).
Why this piece earns its place
A phrase boundary is not handed to you — something has to detect it, on partial transcripts that are themselves still being revised, and that detector is what really governs latency variance. Two consequences get missed. First, the tail: a speaker who trails off, or whose last words never satisfy the boundary rule, leaves a chunk that is never flushed, so the end of that utterance is simply never translated. Average latency looks healthy, because a missing chunk produces no slow measurement — it produces no measurement at all. You need a timeout that force-flushes a stale buffer, and that timeout is part of the policy, not an implementation detail someone adds later. Second, chunk size is not the same dial as context, and they get conflated constantly. Shrink the chunk and the translator does not see less; it sees the same words arriving as history rather than as input, so it re-encodes a growing prefix on every emission and cost per second of speech climbs the longer someone talks without stopping. How much prefix to carry, and when to drop it, is a second decision — and when latency drifts late in a long turn, it is usually that one, blamed on the first. Boundaries also become the unit of recovery — when a stage dies you replay from the last one — so smaller chunks make failure cheaper as well as making output earlier.
Step 5 · Voice-preserving synthesis
Should the translation sound like the speaker?
The translated text needs to become speech. Should the synthesized voice be a generic, neutral voice, or attempt to preserve something of the original speaker's actual voice and delivery?
Synthesizing the translated speech — generic neutral voice, or preserve the original speaker's characteristics?
A fixed generic voice strips away speaker identity and emotional delivery (urgency, warmth, humor) entirely — for many real use cases (a video call, a live-translated talk), that loss meaningfully degrades the experience of hearing someone specific rather than a robotic intermediary.
Voice-preserving TTS carries over pitch, tone, and delivery style from the original speaker, so listeners hear something closer to "this specific person, speaking your language" rather than an anonymous narrator reading a translation — a meaningfully richer, more natural experience for live conversation or presentation use cases.
Voice-preserving synthesis is valuable precisely in LIVE settings (video calls, live interpretation) where speaker identity matters most to the listener — restricting it to pre-recorded content misses the primary use case.
Use voice-preserving Streaming TTS that captures elements of the original speaker's prosody and vocal characteristics, so the synthesized translation sounds like a translated version of that specific person rather than a generic narrator — while still meeting the same streaming, low-latency requirement as every other stage.
Why this piece earns its place
Translated speech is rarely the same length as the source, and that is the problem voice preservation quietly inherits. Preserve the speaker's rate faithfully and the synthesized audio drifts further behind with every sentence, until the translation is still talking after the speaker has stopped. Compress it to stay aligned and you have altered the delivery you went to the trouble of preserving. So the synthesizer needs a rate controller wired to the latency budget, and it needs to be told which to give up — pacing fidelity or synchrony — because it cannot hold both when the two languages disagree about length. Two operational points belong with it. A voice-cloning path on a live pipeline is also a voice-spoofing path, so enrollment, consent and some form of output marking are product requirements, not compliance work bolted on after launch. And plan the fallback: when there is not enough clean audio to model a voice, assign a stable voice per speaker for the session rather than picking one at random, so a listener on a multi-party call is not relearning who is who on every turn.
Step 6 · Filter disfluencies before translating
Real speech is messy
Real spoken language is full of "um," false starts, and self-corrections ("I think— actually, I mean..."). Feeding that directly into a translator produces garbled, unusable output.
Add a Disfluency Filter between ASR and translation that strips filler words and resolves self-corrections (keeping the corrected version, discarding the abandoned false start) before the text reaches translation — which assumes reasonably well-formed input and produces much better results on cleaned text than on raw, messy transcription.
Why this piece earns its place
Deleting words breaks alignment, and alignment is what half this pipeline runs on. Caption timing, the Stabilizer's sense of how much context it holds, and any per-stage latency measurement all key off where in the audio a word sat, so the filter has to emit spans — what it kept, what it removed and from where — not just a cleaner string. The sharper constraint is that ASR revises. The filter therefore runs repeatedly over a prefix that keeps changing, and it has to be stable under that: if correcting one word flips an earlier keep-or-drop decision, the change propagates into a translation revision the Stabilizer then has to absorb, and a single ASR correction becomes a visible rewrite of a whole clause. Make it a function of the committed prefix only. Budget, too, for filler patterns being language-specific — the cheap rule-list version does not transfer, so every new source language quietly ships a new lexicon. Skip the filter on prepared speech; it earns its keep on unscripted, hedging conversation.
What the new pieces do
- Disfluency Filterservice
- Cleans real, messy spoken language — filler words, self-corrections, restarts — before it reaches translation, which assumes reasonably well-formed input.
Step 7 · Budget the latency across every stage
No single stage should eat the whole budget
There's a total acceptable end-to-end latency for the experience to feel real-time. How should that budget be allocated across ASR, translation, and synthesis?
Define an explicit Latency Budget split across stages — e.g. a target of a few hundred milliseconds each for ASR, translation, and synthesis — so no single stage is free to silently consume the whole acceptable delay. Each component is engineered and monitored against its OWN slice of the budget, not just an aggregate end-to-end target that doesn't reveal which stage is actually the bottleneck.
Why this piece earns its place
Write the budget as an average and you will miss how this pipeline actually fails. The stages are serial and stateful, so a chunk that overruns does not cost one slow chunk — it pushes everything behind it back, and because speech keeps arriving at the speaker's pace, the lag never recovers inside that turn. Tail latency here is not a percentile over independent requests; it is accumulated, permanent drift. That makes the missing half of the budget a policy: when a stage is out of time, does it emit the lower-quality result it already has, or block and stay correct? Someone has to own that answer per stage, and it is a different answer for recognition than for synthesis. What a stage split omits is everything that is not a stage. Network time, device audio buffering and the playback jitter buffer all sit inside the listener's experience and outside every box on this diagram, so no team owns them and no dashboard shows them. Every stage can hit its slice and the call can still feel late, and the conversation that follows is always the same one: each team is green, so the problem must be somewhere else. Give the client-side slice a number and a name, even if the name is yours, or the budget only ever covers the part that was easy to measure.
What the new pieces do
- Latency Budgetdata
- The total acceptable end-to-end delay, split explicitly across ASR, translation, and synthesis so no single stage silently eats the whole budget.
Step 8 · The sharp edges
Code-switching and knowing when someone's actually done
Two real-world complications: a speaker who switches languages mid-sentence, and correctly distinguishing a mid-thought pause from the actual end of an utterance.
For code-switching, per-segment language detection (not one fixed language assumption for the whole session) lets the pipeline re-route a switched segment through the correct language pair rather than force-translating it as if it were still the original language. For end-of-utterance detection, use a combination of silence duration, intonation cues, and semantic completeness (does this look like a finished thought) rather than silence alone — a thoughtful pause shouldn't be mistaken for "done speaking" and trigger a premature finalization.
Why this piece earns its place
End-of-utterance is the only place this system commits, and its two errors are not symmetric. Fire early and you truncate a thought the listener has already heard spoken, so the repair arrives as an audible correction. Fire late and you add dead air, which is the exact failure the whole design exists to remove. The signals feeding that call also disagree about how fresh they are. Silence is free and true the instant it happens. Intonation comes off audio frames you already hold. Semantic completeness needs the transcript, so it arrives a stage late and is read from text that may still be revised under it. Combine them and the decision is only ever as current as the slowest input, which means the signal you add to stop premature cuts spends part of the dead air it was added to protect. Log which of the three actually carried a given call, or you will spend a week tuning a threshold that was not the thing deciding. On code-switching, the trap is trusting a classifier on a very short segment. Detection over a phrase is unreliable, so weight it with a prior: the session's declared pair, and what this speaker has done so far. A confidently wrong route is debugged in the wrong place, because it looks like a bad translation rather than a routing decision.
You did it
You just designed a realtime translation system.
Everything you assembled, in order
- Waiting for a complete utterance stacks additive latency across every stage — real conversational pacing requires the whole pipeline to stream.
- Streaming ASR emits continuously-updating partial transcripts rather than one final answer, unlocking everything downstream.
- Incremental translation must be willing to REVISE its own output as more context arrives — word order genuinely differs across languages.
- Chunking at phrase/clause boundaries balances context quality against latency, between word-by-word and full-sentence extremes.
- Voice-preserving synthesis carries speaker identity and delivery through translation, not just literal words.
- A disfluency filter cleans messy real speech before it reaches translation, which assumes reasonably well-formed input.
- An explicit per-stage latency budget gives each component a concrete target, rather than one opaque aggregate goal.
- Code-switching and end-of-utterance detection both need dedicated handling beyond a single fixed-language, silence-only assumption.