The 500 millisecond problem
Why translation quality stopped being the bottleneck years ago, and why almost nobody has solved the thing that replaced it.
By BabelGO Engineering
Ask anyone why real-time translation is not solved and you will hear something about accuracy. It is the intuitive answer, and it has been wrong for about three years.
Modern translation models handle ordinary business conversation well enough that quality is rarely what makes someone abandon a tool. What makes them abandon it is the pause.
The pause is the product
Human conversational turn-taking runs on a gap of roughly 200 milliseconds between one speaker finishing and the next starting. It is remarkably consistent across languages and cultures. Stretch that gap past about half a second and something perceptual flips: the exchange stops feeling like a conversation and starts feeling like a transaction with a machine in the middle.
Most translation tools land between two and four seconds. That is not a worse conversation — it is a different activity.
Where the seconds go
A naive pipeline is four sequential blocking steps:
- Wait for the speaker to stop.
- Transcribe the complete utterance.
- Translate the complete transcript.
- Synthesise the complete translation.
Each step waits for the previous one to finish. The waiting, not the compute, is where the budget disappears. Voice activity detection alone typically burns 600–800 ms deciding that a speaker has actually stopped rather than paused for breath.
Streaming everything
The fix is conceptually simple and operationally miserable: never wait for a complete anything.
- Recognition emits partial hypotheses every few frames.
- Translation consumes partials and speculatively decodes ahead, revising when the source revises.
- Synthesis begins on the first stable clause rather than the final token.
This means every stage must tolerate its input changing underneath it. A translation committed at 200 ms may need to be revised at 340 ms because the speaker added a negation. Handling that gracefully — without the synthesised voice audibly stuttering or contradicting itself — is most of the engineering.
What we measure
We track a single number end to end: the interval between the acoustic end of a source utterance and the first sample of translated audio. Not model inference time, not time-to-first-token. The number a person in the room would measure with a stopwatch.
Our current median across supported pairs is 420 ms. The p95 is 680 ms, which is still too high, and it is what most of this quarter goes into.
If you only optimise the median, you ship something that feels magical in a demo and unreliable in a hall.
What is left
The remaining latency is not evenly distributed. Roughly 40% of it sits in translation, and most of that in language pairs with heavy word-order divergence — Japanese to English being the canonical pain. When the verb arrives last, speculative decoding has less to work with and revisions get more expensive.
That is a research problem, not an infrastructure one, and it is the honest reason a handful of pairs are slower than the rest.