The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
A batch problem with its own hard constraints
Dubbing isn't a live conversation — there's no round-trip latency pressure. But it has its own genuinely hard constraints: the translated speech has to fit the ORIGINAL video's timing, needs to look right against the original mouth movements, and — because it's recreating a real person's voice and face — carries real consent and ethical weight most content-processing pipelines never have to think about.
We'll build for those specific constraints: timing-constrained (not latency-constrained) translation, offline-quality voice cloning, a real choice about how to handle lip-sync, cross-segment consistency, and consent as a first-class gate, not an afterthought.
What the new pieces do
- Creatorclient
- Has existing video content and wants it genuinely watchable in another language — not just technically translated.
Step 1 · The baseline
Translate and overlay, ignoring timing
Simplest version: transcribe the original speech, translate the text, synthesize it, and overlay it onto the video — replacing the original audio track. What goes wrong?
Translated speech is synthesized and simply overlaid onto the original video. What's the problem?
Translation accuracy alone doesn't address a completely separate problem: different languages take different amounts of time to say the same thing, and ignoring that means the dub doesn't just sound off, it visibly and audibly drifts out of sync with the video.
A sentence that takes 3 seconds in one language might take 4 or 2 in another — without accounting for that, the translated audio doesn't line up with when the original speaker's mouth is actually moving, and the mismatch compounds across a whole scene, becoming increasingly obvious and distracting.
Timing drift begins accumulating from the very first mismatched line, not only in long content — a short clip with even a couple of poorly-timed lines already shows the problem clearly.
Languages differ in verbosity — a translated line is rarely the exact same length as the original — so naive overlay causes the dubbed audio to drift out of sync with the video's timing and visible mouth movements, a mismatch that compounds and becomes increasingly noticeable across a scene.
Why this piece earns its place
Build the naive version anyway, because the artifact it produces is the one everything downstream depends on: a table of segments, each with a speaker, a start time and an end time. Those boundaries are a commitment. Re-segment later — merge two lines, split a long one — and every translation, every synthesized take and every human sign-off attached to the old boundaries is orphaned, so segment ids want to be stable and derived from the source, not from position in a list. The drift also arrives in two shapes, and only one of them is catchable. Let each dubbed line run on and the audio simply outlasts the picture, which a total-duration comparison against the source catches — the cheapest check in the pipeline and worth wiring before anything else. Anchor each line to its original start timestamp instead and the file comes back exactly as long as the source, because the overrun has been absorbed as clipped tails and pockets of silence inside it. That variant is green everywhere: the job succeeds, the durations agree, no stage reports an error, and the only remaining detector is a person watching playback. Which is why the transcript is worth storing as a first-class editable artifact: a wrong word there is the cheapest thing in the pipeline to fix by hand and the most expensive to leave.
What the new pieces do
- Video Ingestgateway
- Accepts existing, already-recorded video — this is offline content processing, with no real-time latency pressure, unlike a live conversation.
- Transcribe + Diarizeservice
- Transcribes the original speech and identifies which speaker said what and when — a prerequisite for dubbing multiple speakers correctly.
Step 2 · Translate for timing, not just meaning
A different constraint than realtime translation
Fix Step 1: translation needs to fit roughly the same time window as the original line. How is this different from the latency problem in realtime speech translation?
Dubbing translation needs to fit a specific TIME DURATION. How is this different from realtime translation's core challenge?
Realtime translation optimizes for minimizing LATENCY (how fast can output start appearing). Dubbing has no such pressure — it's offline, batch content — but instead needs the FINAL translated line to fit a specific target DURATION, a genuinely different optimization goal.
This is a different axis entirely from realtime translation's speed pressure — dubbing can take as long as it needs to PRODUCE a translation, but that translation then has to satisfy a duration constraint (fit the original timing) that realtime translation never has to solve, since realtime translation just needs to keep pace, not match a pre-existing time window.
Timing is the SPECIFIC differentiating constraint being discussed here — accuracy matters in both cases, but it doesn't explain what makes dubbing's translation problem genuinely different from realtime's.
Build Timing-Aware Translation that treats the original segment's duration as a real constraint — paraphrasing, compressing, or expanding the translation as needed so the spoken result roughly fits the same time window, not just conveys the same meaning.
Why this piece earns its place
Duration is not a property of text, which means this stage cannot verify its own output. Words are not seconds — you only know whether a line fits after something has said it. So translation becomes a generate, measure, retry loop with a duration estimator or the synthesizer itself inside it, not a single call, and that couples two stages the diagram draws as separable. Put a durable queue between them and the loop does not become impossible, only explicit: the retry needs a per-segment state machine carrying the attempt count, the last measured duration and why a take was rejected, because nothing holds that in memory any more. In-process retry gives you all of that free, and takes it back the first time a job has to survive a restart. Two knobs decide how often the loop runs, and both belong on the segment, not in global config. The first is the tolerance band: how far over or under the original a take may land before rejection. The second is whether speaking rate may be adjusted — the escape hatch a human dubbing director reaches for, and one with a ceiling past which the line stops sounding like a person. Tolerance is not uniform either: a line that ends on a cut, or on a beat before anyone speaks again, can borrow time from the silence around it; one that must finish before the next speaker starts cannot.
What the new pieces do
- Timing-Aware Translationservice
- Translates for meaning AND for pacing — the translated line needs to roughly fit the same time window as the original, not just mean the same thing.
Step 3 · Higher-quality voice cloning, offline
More compute budget, no latency pressure
Synthesize the translated line in a voice resembling the original speaker. Since this is offline batch processing rather than a live conversation, does that change anything about the approach?
Voice cloning for dubbing happens offline, not in real time. Does that change the approach compared to a live setting?
Reusing a latency-optimized approach when latency isn't actually a constraint here leaves real quality on the table — offline processing can afford a meaningfully more expensive, higher-fidelity synthesis approach since there's no live conversational pace to keep up with.
The absence of a real-time constraint is a genuine opportunity: offline dubbing can afford slower, more thorough voice synthesis techniques that produce meaningfully better vocal fidelity than what's practical when speed is a hard requirement — the quality bar can simply be higher.
Speed and quality are frequently in real tension for generative synthesis — removing the speed constraint genuinely opens up higher-quality options that a latency-bound system can't use.
Because dubbing has no real-time latency pressure, use a more computationally expensive, higher-fidelity Voice Cloning approach than would be viable for live conversation — offline processing affords quality that a latency-bound system genuinely can't.
Why this piece earns its place
More compute is the boring half of this. The useful half is that being offline lets you generate several takes and keep the best one — which is the same budget the timing loop is already spending — so budget them together, per segment, instead of giving each stage its own fixed allowance. Two things set the ceiling. The first is reference audio: a clone is only as good as the cleanest stretch of that speaker alone you can find, and in real content the speaker is rarely alone, which quietly makes speech separation an input to clone quality rather than only to the final mix. Diarization is what lets you assemble that reference per speaker at all. The second is reproducibility. Sampling-based synthesis will not hand you the same take twice, so store the rendered audio as the artifact and pin the seed beside it. A reviewer approving a dub is approving one specific waveform; if a later re-run regenerates instead of replaying it, the approval covered something that no longer exists.
What the new pieces do
- Voice Cloningservice
- Synthesizes the translated line in a voice resembling the original speaker — with real compute budget available, since this is offline, not real-time.
Step 4 · The lip-sync trade-off
Fit the audio to the video, or the video to the audio?
The dubbed audio needs to align with the speaker's mouth movements. There are two genuinely different approaches: constrain the AUDIO's timing/phonemes to match the EXISTING video, or actually modify the VIDEO's mouth movements to match the new audio. Which is the right default?
Two lip-sync approaches exist: fit audio to the existing video, or modify the video's mouth movements to fit the audio. Which should be the default?
Audio-to-video fitting is less invasive — it doesn't alter the original recorded footage of a real person at all — while still producing a reasonably synchronized result through careful timing and phoneme-aware synthesis. Modifying someone's actual recorded mouth movements is a form of video manipulation with real consent, authenticity, and deepfake-adjacent concerns that shouldn't be a casual default.
Defaulting to actually altering a real person's recorded video, even for a legitimate dubbing purpose, raises genuine consent and authenticity concerns that shouldn't be the unconsidered default — visual seamlessness isn't the only value at stake here.
They are NOT equivalent — one leaves the original video footage completely unaltered, the other directly manipulates a real, identifiable person's recorded likeness, which is a materially different, more ethically weighted intervention.
Default to fitting audio to the existing video — careful, timing-aware synthesis and phoneme alignment against the unaltered original footage — rather than directly modifying the video's recorded mouth movements. Video manipulation carries genuinely different consent and authenticity weight than adjusting synthesized audio, and shouldn't be the unexamined default.
Why this piece earns its place
Make this a per-segment decision rather than a pipeline-wide one. Lip-sync only buys anything where a mouth is legible — speaker on camera, facing it, close enough to read — and in most real footage that is a minority of the runtime. Without a per-shot signal for it you run the expensive stage uniformly and get nothing back across the cutaways, the wide shots and the voice-over. With one, lip-sync becomes conditional and the cost stops scaling with the length of the video. What is legible is narrower than the whole line, too. The closures on m, b and p and the wide-open vowels are the only positions a viewer can really read; between them it is mush at any ordinary viewing distance. So alignment is worth scoring at those positions and nowhere else — a line can sit loose across its middle and still look right. Dubbing practice adds a separate convention on top: match the frame where the mouth opens and the frame where it closes, whatever happens in between. That one the translator can honour directly, since it constrains where a line starts and stops rather than what it contains. Keep the dubbed speech as its own track and the composited render as a derived artifact. Then a re-dub is a re-mix. Commit the sync into the picture instead and every later change is a re-render, from an original you now have to remember to keep.
What the new pieces do
- Lip-Syncservice
- Aligns the dubbed audio to the video's mouth movements — a real design choice between two genuinely different approaches with different ethical weight.
Step 5 · Consistency across a long video
A voice — and a term — shouldn't drift scene to scene
A feature-length video is processed in many segments for parallelization. Should each segment be translated and voiced completely independently?
Segments are processed independently for parallelization. Should terminology and voice stay independent per segment too?
Fully independent processing risks a term translated one way in scene 3 being translated differently in scene 20, or a character's cloned voice subtly drifting between segments — both genuinely break the viewing experience even if each individual segment is locally fine.
A shared consistency reference lets segments be processed in parallel (for throughput) while still drawing on the SAME established terminology and voice choices, so the final result reads and sounds consistent across the whole video, not just within any one segment.
Both terminology drift and voice drift are real, independent consistency risks across a long video — addressing only one leaves the other genuinely unresolved.
Maintain a shared Cross-Segment Consistency store for terminology and voice choices — parallelized segments read from and contribute to it, so translation and voice stay consistent across the whole video even though individual segments are processed independently for throughput.
Why this piece earns its place
Shared mutable state plus parallel workers is a write-ordering problem, and that is the part that bites. Segments are the only things that ever meet a new term, so they have to contribute — but if a contribution is a plain write landing the moment a worker reaches it, segment twenty can finish ahead of segment three and set the term first. The result then depends on how the queue happened to schedule the run, and dubbing the same video twice gives two different videos. So make a contribution a claim rather than a write: keyed by the source term, stamped with the segment index it came from, and resolved so the earliest position in the video wins regardless of which worker got there first. A late claim from segment three displaces segment twenty's, and the few segments already rendered against the loser get re-rendered — affordable, because it is those segments and not the video. Do not put both kinds of state in one store either. A speaker's voice reference has to be pinned and identical everywhere; the glossary gets edited while the video is still being reviewed. Conflate them and a wording fix invalidates audio that was fine. And the glossary holds more than nouns — forms of address and formality register are decisions for the whole runtime, and they are what a viewer notices first when they slip.
What the new pieces do
- Cross-Segment Consistencydata
- Tracks voice and terminology choices across a long video's many segments, so a term or a character's voice doesn't subtly drift scene to scene.
Step 6 · Consent for voice likeness
This recreates a real, identifiable person's voice
Voice cloning synthesizes speech in a real person's actual voice. What needs to be true before that's allowed to happen?
Require a verified consent chain before voice cloning proceeds — technical capability to clone a voice doesn't establish authorization to do so. Attach provenance metadata (or watermarking) to dubbed, synthesized audio, disclosing that it's AI-generated rather than the original recording. This fails closed: without verified consent, the system refuses to clone the voice, regardless of how the request is framed.
Why this piece earns its place
Consent is not a boolean on a speaker, and modelling it as one is the bug. It is scoped — to a set of languages, to a piece of content, to a window of time — so the gate has to evaluate the record against this job rather than ask whether the person is on a list. A record that says yes with no scope will cheerfully authorize the next project too. The part that decides your data model is revocation, because it reaches backwards. When a speaker withdraws, the question is not whether to stop; it is which already-delivered assets came from that voice. You can only answer that if the consent id was stamped onto every synthesized take at the moment it was generated, alongside the seed and the reference audio. Bolt it on afterwards and the honest answer is a full re-scan of the library. It is also why disclosure has to live in two places. Container metadata is stripped by the first re-encode anyone performs — a platform transcode, a clip, a re-upload — so the signal that survives contact with the internet is the one carried in the audio itself.
Step 7 · Scale review by content stakes
Not every dub needs the same scrutiny
A small creator's casual video and a major theatrical release both need dubbing. Should both go through identical review before release?
Route content through a Review Gate scaled by content stakes — audience size, commercial value, sensitivity — rather than one uniform policy: lower-stakes content can ship fully automated, while high-stakes content (a major release) gets human review and adjustment of the automated dub before final delivery. This matches review rigor to actual consequence, similar in spirit to reversibility-scoped gates elsewhere, but scoped here by the content's reach and stakes rather than the action's undoability.
Why this piece earns its place
Whether there is a reviewer matters less than what the reviewer is handed. Review is the only stage here whose cost is a person's attention measured against the video's own runtime, and it is paid again for every language shipped — while everything upstream is compute that can be run overnight or run wider. A gate that means somebody sits through the whole thing therefore spends the saving the automation was bought for, which is why that gate is the one that quietly stops being kept. Make it targeted instead: route the reviewer to the segments the pipeline was least sure of — where the timing loop burned its retries, where the take landed at the edge of the tolerance band, where the translation had to compress hardest, where a speaker label was ambiguous. That only works if the earlier stages emit a per-segment effort signal, which is something you have to decide to build before you need it. Then make the verdict actionable. A note in a spreadsheet is not a fix — a reviewer's edit should re-enter the pipeline at the stage it belongs to and re-render that one segment, which is affordable precisely because segment ids were stable from the first step. And default the classification to the strict side: content whose stakes are unknown gets reviewed, or every misclassification fails open.
What the new pieces do
- Review Gateservice
- Routes high-stakes content (a major release) through human review before final delivery; low-stakes content can ship fully automated.
Step 8 · The sharp edges
Don't lose the soundtrack, and know who's speaking
Two more real complications: the original audio track usually contains background music and sound effects alongside speech, and scenes with multiple speakers or overlapping dialogue need to know who said what.
Cleanly separate the speech track from background music and effects during processing (rather than replacing the ENTIRE audio track and losing the soundtrack), and re-mix the dubbed speech back in alongside the preserved background audio. For multi-speaker scenes, rely on the diarization from Step 1 (who said what, when) to dub each speaker's lines correctly and distinctly, and flag the spans where speech genuinely overlaps instead of assuming both lines can be dubbed cleanly.
Why this piece earns its place
The thing worth knowing about separation is where its mistakes end up. You keep the residual — whatever the separator declined to call speech — so anything it misses stays audible underneath the dub: a breath, the tail of a word, a line quiet enough to read as room tone. The result is the original language faintly present beneath the new one, obvious on headphones and inaudible on a laptop speaker, which is to say it survives exactly the check most teams actually run. The loss runs the other way as well, and it lands on music that was playing under the dialogue. Which makes the first question whether you need to separate at all. A professionally finished production usually ships an M&E stem — music and effects with the dialogue already absent — built for this exact purpose. If the source has one, take it and delete the stage. Overlapping dialogue is the case with no clean answer: two people talking over each other cannot become two clean dubbed lines, so someone has to decide which one carries the scene, and that someone is a person.
You did it
You just designed an AI video dubbing system.
Everything you assembled, in order
- Naive translate-and-overlay ignores that languages differ in verbosity — the dub drifts out of sync almost immediately.
- Timing-aware translation is time-budget-constrained (fit the original duration), a genuinely different problem from realtime translation's latency constraint.
- Offline processing affords higher-quality voice cloning than a latency-bound live system could use.
- Fitting audio to the existing, unaltered video is the more conservative default versus directly modifying a real person's recorded mouth movements.
- Cross-segment consistency keeps terminology and voice coherent across a long video processed in parallelized chunks.
- Voice cloning requires a verified consent chain and provenance disclosure — capability to clone a voice doesn't establish authorization.
- Review rigor scales with content stakes, not one uniform policy for every piece of content.
- Background music/effects are preserved separately from the dubbed speech track, and diarization correctly attributes multi-speaker dialogue.