Real-Time Speech Translation Latency for Voice Agents

Real-time multilingual translation feels like one capability, but under the hood it is a chain of models each spending milliseconds you cannot get back. This guide is a latency teardown: where the milliseconds actually go, what round-trip number to target, and why your pipeline topology matters more than any single model swap.

The short answer: what "real-time" costs you

The historical bar was ugly. Classic cascaded speech-to-speech systems chained ASR, machine translation, and TTS and incurred significant delays of 4-5 seconds, forcing stilted, turn-based conversation. That era is over.

Today there are two realistic targets depending on your architecture:

• End-to-end speech-to-speech: Google has shown live translation in the original speaker's voice with only a 2-second delay.

• Well-engineered cascaded: Deepgram reports a cascaded ASR-MT-TTS pipeline can stream under 500ms total latency.

For a live multilingual voice agent, treat sub-500ms round-trip as the aspirational ceiling and sub-3 seconds as the "conversation still feels live" floor. Below, we break that budget down stage by stage.

The latency budget, stage by stage

The mental model that matters: in a naive sequential pipeline your total latency is the sum of every stage. In a streaming-parallel pipeline, stages overlap and your total approaches the longest stage plus handoff overhead. Everything in this section is about collapsing a sum into a max.

Audio capture, buffering & VAD

Before a single model runs, you buffer audio and decide when the speaker has paused. Use 16 kHz as your default sample rate - Silero VAD, WebRTC VAD, and most production STT services converge on that standard, per Deepgram's streaming guidance. Endpointing is its own tunable latency; we cover turn-taking in depth in our end-of-turn detection teardown.

ASR / streaming STT

This is the single biggest lever. Switching from offline to streaming ASR cuts latency by roughly 9x for 60-second inputs, because you emit partial transcripts as audio arrives instead of waiting for silence. The transport and GPU-side mechanics of doing this well are in our dedicated guide on streaming STT latency over WebSockets on serverless GPU.

MT: the multilingual tax

Translation is where multilingual agents pay extra. You cannot translate a half-formed clause reliably, so MT either waits for more context (adding delay) or translates speculatively (risking revisions). Low-resource language pairs and script differences compound both the latency and the error rate - we walk through the practical failure modes in overcoming transcription challenges for multilingual voice agents.

TTS: the compute bottleneck

TTS is the dominant cost. Deepgram measures TTS at 62% of total pipeline compute time. The metric that saves you is time-to-first-byte: advanced streaming TTS can report under 25ms P99 TTFB server-side. On our own infrastructure, we measured a ~100ms TTFB on the Orpheus TTS streaming endpoint, which is comfortable for conversational voice. Picking a first-byte-optimized voice is covered in our OpenAI TTS alternatives comparison.

Network overhead

Every hop between client, GPU, and third-party TTS adds round-trip time. Co-locating stages and keeping audio frames on persistent connections matters; see network latency in the AI inference pipeline.

Cascaded vs end-to-end (speech-to-speech): the real tradeoff

Neither architecture is categorically faster. A tuned cascade can beat a monolithic model because each stage streams independently, while end-to-end models remove handoff overhead and preserve prosody. The decision is nuanced enough that we gave it its own article: speech-to-speech vs cascaded voice agent latency. The short version: cascaded gives you control and swappable components; end-to-end gives you the original speaker's voice and fewer seams.

Simultaneous translation: translating before the sentence ends

The frontier is simultaneous translation - emitting target-language audio before the source sentence finishes, the way a human interpreter does. This trades a little quality for a large perceived-latency win, and it changes how you must measure.

Measuring it right: Average Token Delay (ATD)

Commercial latency vocabulary (TTFB, round-trip) misses something simultaneous systems care about: when each partial translation lands. The research community proposes Average Token Delay (ATD), a metric that focuses on the end timings of partial translations rather than just when translation begins. If you are building simultaneous translation, report ATD alongside wall-clock - a system that starts fast but revises constantly will look great on first-token latency and terrible on ATD.

Streaming optimizations that actually move the budget

• Streaming ASR - the ~9x win above; non-negotiable for real time.

• Chunked ASR with the right chunk size - smaller chunks lower latency but raise compute and error risk.

• Async queue buffering between stages so ASR, MT, and TTS run concurrently instead of blocking each other.

• TTS first-byte streaming so audio plays while the rest of the sentence is still synthesizing.

• Context flushing to manage memory on long sessions.

• Perceived-latency tricks like filler phrases that mask retrieval or model time.

Realistic targets & benchmarks

These are measured numbers, not marketing. Several are Cerebrium's own benchmarks on serverless GPU infrastructure.

From our testing, the Ultravox + Cartesia stack hit 600ms end-to-end first-time-to-audio on our serverless infra. On the LiveKit commentator pipeline we measured ~500ms model frame interpretation plus ~180ms Cartesia voice generation, and the filler technique brought perceived time-to-first-audio to ~1.0s. At the opposite end of scale, cascaded multilingual translation has run sub-three-second end-to-end at 22,000 participants across six languages.

Where you host the LLM also moves the number: self-hosting on Cerebrium's serverless infrastructure is reported at 50% faster latency and 3x cheaper versus a hosted realtime API. If you are assembling the whole thing, our Pipecat voice agent GPU deployment guide is the practical starting point.

How to measure your own pipeline

1. Instrument per-stage spans. Capture a timestamp at every handoff: audio-in, VAD decision, ASR partial, ASR final, MT out, TTS first byte, audio-out. You cannot optimize a budget you cannot see.

2. Report p50 and p95 wall-clock, not averages. Tail latency is what breaks a conversation.

3. Add ATD for simultaneous systems. Pair it with first-token latency so you catch revision churn.

4. Separate processing latency from network latency so you know whether to optimize models or transport.

5. Watch concurrency. Latency under load is the real test; see concurrent voice sessions per GPU.

Collapse the sum into a max, stream every stage, measure the tail - and a multilingual voice agent that once felt like a walkie-talkie starts to feel like a conversation.

Frequently asked questions

What is an acceptable latency for a real-time multilingual voice agent?

Treat sub-500ms round-trip as the aspirational ceiling for a well-engineered cascaded pipeline, which Deepgram reports is achievable, and sub-3 seconds as the floor where conversation still feels live. The old 4-5 second cascaded delay is no longer competitive.

Is end-to-end speech-to-speech faster than a cascaded pipeline?

Not categorically. A tuned cascade can match or beat end-to-end because each stage streams independently, while end-to-end models remove handoff seams and preserve the speaker's voice. Google has shown end-to-end at a 2-second delay; the tradeoff is control versus voice fidelity.

What is Average Token Delay (ATD)?

ATD is a latency metric for simultaneous translation that focuses on the end timings of partial translations rather than just when translation begins. It catches systems that start fast but revise constantly, which first-token latency alone would miss.

Why does multilingual translation add latency?

Machine translation needs enough source context to translate reliably, so it either waits (adding delay) or translates speculatively (risking revisions). Low-resource language pairs and differing scripts compound both latency and error rate, which is the multilingual tax on top of ASR and TTS.