The big idea

A conversation, not a walkie-talkie

Chain speech-to-text, translation, and text-to-speech together and you have SOMETHING — but "wait for the speaker to finish, then translate the whole thing, then speak it" produces a multi-second lag per exchange that makes a real back-and-forth conversation feel like a walkie-talkie: talk, wait, listen, wait, repeat.

We'll build it to actually stream: partial transcripts, incremental translation that revises itself as more context arrives, careful chunking, voice-preserving synthesis, and an explicit latency budget split across every stage.

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the diagram grow, hover the boxes, and at the end kill translation and mid-sentence code-switch to see what degrades and what needs re-routing. Hit Begin.

Step 1 · The baseline

Record, then translate, then speak

Simplest version: wait for the speaker to finish a full utterance, transcribe it, translate the complete text, synthesize speech, then play it. What's the actual cost of waiting for completeness?

Design decision: The pipeline waits for a full utterance before doing anything. What's the cost?

The call: End-to-end latency stacks the full length of the utterance plus all three processing stages before the listener hears anything, making the exchange feel sequential and stilted rather than like an overlapping, real conversation. — Waiting for completeness means the listener's first sound arrives only after the ENTIRE pipeline has processed the ENTIRE utterance — for anything but very short sentences, that delay breaks the natural overlap and pacing of real conversational turn-taking.

Waiting for a complete utterance stacks its full length plus every stage's processing time before the listener hears anything — for real conversational pacing, that delay is disqualifying. The entire pipeline needs to stream, producing partial output as input arrives, not batch-process complete utterances.

Latency compounds across a pipeline: Three sequential stages, each waiting for the previous one's COMPLETE output, means total latency is additive across all three plus the input's own length — streaming is what breaks that chain, letting stages work on partial input concurrently instead of in strict sequence.

Step 2 · Streaming ASR

Partial transcripts, not one final answer

Fix the biggest lag source first: don't wait for the speaker to finish before transcribing.

Design decision: Should transcription wait for the speaker to pause or finish, or produce output continuously?

The call: Emit partial transcripts continuously as speech arrives, updating and refining them (correcting earlier words if needed) as more audio comes in. — Streaming ASR treats transcription as a continuously-updating process rather than a single final answer — early partial text may be revised as more context arrives, but something is always available to feed downstream immediately, which is what unlocks the rest of the pipeline streaming too.

Make ASR streaming: it continuously emits partial transcripts as audio arrives, revising earlier words if later context changes the most likely interpretation. This gives downstream translation something to work with immediately, rather than only after the speaker stops.

Partial output that can be revised beats waiting for certainty: A streaming system trades "always exactly right the first time" for "immediately available, occasionally revised" — for a real-time pipeline, having SOMETHING to act on now, even if it might be corrected shortly, is usually the better trade than silence until certainty.

Step 3 · Incremental translation must revise itself

Word order isn't the same across languages

Translating a partial sentence before it's complete forces the translator to guess. In German, the main verb often comes at the END of a clause — translating the first few words into English before seeing that verb can require a fundamentally different sentence structure than what was guessed. What should happen?

Design decision: A partial translation was based on an early guess that later context contradicts. What should happen?

The call: Allow the translation to be RE-REVISED as more context arrives — a Stabilizer re-translates the segment when new words meaningfully change the most likely correct translation, updating the displayed/spoken output. — Because early-arriving words genuinely may not determine the correct translation until more context (like a German sentence-final verb) arrives, the system needs to treat its own partial output as provisional and revise it — the alternative is either high latency (wait for certainty) or wrong output (never revise).

Add a Re-translation Stabilizer: as more of the sentence arrives, it re-evaluates whether the current translation is still the best one and revises the displayed or spoken output if a meaningfully better translation is now possible — accepting a degree of self-correction as the necessary cost of low latency, rather than either waiting for certainty or locking in an early wrong guess.

Streaming translation is inherently provisional: Unlike streaming ASR (which mostly refines transcription of what was actually SAID), streaming translation must sometimes fundamentally restructure its output because a language's grammar genuinely doesn't reveal enough information until later in the sentence — this is a deeper, language-structural challenge, not just an engineering optimization.

Step 4 · Chunking policy

Not word-by-word, not full-sentence

How large should each chunk of text handed to translation be — every single word the instant it's transcribed, or wait for a reasonably complete phrase?

Design decision: What's the right granularity for feeding transcribed text into the translator?

The call: Chunk at natural phrase or clause boundaries — enough words to give the translator meaningful context, without waiting for the entire sentence — balancing translation quality against latency. — Phrase-level chunking gives the translator enough local context to produce a reasonably confident translation while still emitting output well before the full sentence completes — a deliberate middle ground between the two extremes, tuned to the actual latency/quality trade-off.

Chunk at natural phrase or clause boundaries — enough accumulated context for a reasonably confident translation, without waiting for full-sentence completeness. This is a tunable policy sitting deliberately between word-by-word (too little context, too much revision) and full-sentence (too much latency).

Chunk size is a latency/quality dial: Every streaming system has some version of this trade-off — smaller chunks reduce latency but increase revision frequency and reduce per-chunk context; larger chunks improve quality per chunk but increase latency. There's no universally correct size, only a tuned trade-off for the specific use case.

Step 5 · Voice-preserving synthesis

Should the translation sound like the speaker?

The translated text needs to become speech. Should the synthesized voice be a generic, neutral voice, or attempt to preserve something of the original speaker's actual voice and delivery?

Design decision: Synthesizing the translated speech — generic neutral voice, or preserve the original speaker's characteristics?

The call: Use voice-preserving synthesis that captures characteristics of the original speaker's voice and prosody, so the translation sounds like a translated version of THAT PERSON, not a generic narrator. — Voice-preserving TTS carries over pitch, tone, and delivery style from the original speaker, so listeners hear something closer to "this specific person, speaking your language" rather than an anonymous narrator reading a translation — a meaningfully richer, more natural experience for live conversation or presentation use cases.

Use voice-preserving Streaming TTS that captures elements of the original speaker's prosody and vocal characteristics, so the synthesized translation sounds like a translated version of that specific person rather than a generic narrator — while still meeting the same streaming, low-latency requirement as every other stage.

Fidelity extends beyond the words: A translation that's lexically perfect but delivered in a flat, generic voice loses real information — tone, emphasis, emotion — that was part of the original message. Voice preservation treats delivery as part of what's being translated, not just the text.

Step 6 · Filter disfluencies before translating

Real speech is messy

Real spoken language is full of "um," false starts, and self-corrections ("I think— actually, I mean..."). Feeding that directly into a translator produces garbled, unusable output.

Add a Disfluency Filter between ASR and translation that strips filler words and resolves self-corrections (keeping the corrected version, discarding the abandoned false start) before the text reaches translation — which assumes reasonably well-formed input and produces much better results on cleaned text than on raw, messy transcription.

Clean the input before the stage that assumes clean input: Rather than making the translation model itself robust to every possible disfluency pattern, a dedicated upstream filter handles this domain-specific cleanup once, keeping the translation stage focused on what it does best.

Step 7 · Budget the latency across every stage

No single stage should eat the whole budget

There's a total acceptable end-to-end latency for the experience to feel real-time. How should that budget be allocated across ASR, translation, and synthesis?

Define an explicit Latency Budget split across stages — e.g. a target of a few hundred milliseconds each for ASR, translation, and synthesis — so no single stage is free to silently consume the whole acceptable delay. Each component is engineered and monitored against its OWN slice of the budget, not just an aggregate end-to-end target that doesn't reveal which stage is actually the bottleneck.

An aggregate target hides which stage is the problem: "Keep total latency under 800ms" doesn't tell any individual team or component what THEY need to hit — splitting the budget per stage makes each one's target concrete and gives clear ownership when the aggregate target is missed.

Step 8 · The sharp edges

Code-switching and knowing when someone's actually done

Two real-world complications: a speaker who switches languages mid-sentence, and correctly distinguishing a mid-thought pause from the actual end of an utterance.

For code-switching, per-segment language detection (not one fixed language assumption for the whole session) lets the pipeline re-route a switched segment through the correct language pair rather than force-translating it as if it were still the original language. For end-of-utterance detection, use a combination of silence duration, intonation cues, and semantic completeness (does this look like a finished thought) rather than silence alone — a thoughtful pause shouldn't be mistaken for "done speaking" and trigger a premature finalization.

Design for the unhappy path: Translation stage down → ASR/captions keep working. Mid-sentence code-switch → per-segment language re-detection. Ambiguous pauses → multiple signals, not silence alone. A translator that only works for one speaker, one language, speaking in clean, complete sentences is a demo; one that handles real, messy, mixed-language conversation is a product.

You did it

You just designed a realtime translation system.

• W — a
• S — t
• I — n
• C — h
• V — o
• A —
• A — n
• C — o