End-of-Turn Detection Latency in Voice Agent Pipelines

Most teams optimizing a voice agent chase the obvious numbers: time-to-first-token on the LLM, streaming TTS latency, GPU cold starts. Then they ship, and the agent still feels sluggish. The culprit is usually the component nobody profiled: end of turn detection model latency in voice agent pipelines. Endpointing is the latency you're not measuring, and it sits at the very front of the chain.

Why endpointing latency dominates perceived response time

End-to-end latency in a voice agent is "the time between when a user stops speaking and when the agent starts responding." Turn detection is the first domino: nothing downstream can start until endpointing commits to the decision that the user is done.

That is what makes it structurally different from every other latency line item. A slow LLM only adds to turns where the model is the bottleneck; a slow endpointer adds to every single turn, before STT finalization, before the LLM, before TTS. "Endpointing latency is how quickly the model decides someone is done speaking" - and it is spent as dead air the caller feels directly.

Endpointing vs VAD vs turn detection: getting the terms right

These terms get used interchangeably and shouldn't be.

• VAD (Voice Activity Detection) answers a frame-level question: is there speech in this audio chunk right now? Silero VAD is the near-universal default - it processes 30ms chunks in under 1ms on a single CPU thread.

• Endpointing answers a turn-level question: has the speaker finished their turn? It typically consumes VAD output plus a silence timer.

• Turn detection / end-of-turn (EOT) / end-of-utterance (EOU) is the broader decision, increasingly made by semantic models that read meaning, not just silence. "Semantic VAD" is the same idea under a different vendor's name.

They compose: VAD feeds endpointing, and endpointing is the specific stage whose latency you budget for. Confusing them is why the latency hides - teams benchmark VAD's sub-millisecond frame cost and assume the turn decision is free.

Sizing the problem: the ~800ms latency budget

The ceiling is brutal. "Eight hundred milliseconds is... roughly all the time a voice agent has before a caller feels something is wrong." Within that budget, "endpoint detection spends its budget on purpose: it waits to be sure you stopped talking, and every millisecond of that silence timer adds straight to response time." A fixed silence timer of 200-400ms can eat a third to a half of your entire budget before the LLM has seen a single token.

For context on how little slack remains, a pause of even 400 milliseconds after you stop talking starts to feel laggy on a phone call. Humans leave gaps of only a couple hundred milliseconds between turns, so the perceptual bar is set well below your technical budget.

Approaches to end-of-turn detection

1. Fixed silence timeout. Wait N milliseconds of silence, then fire. Simple, but it forces a direct choice: short timers cut people off mid-thought, long timers feel laggy.

2. VAD-driven endpointing (e.g. Silero). Cheap and fast at the frame level, but still ultimately gated by a silence heuristic for the turn decision.

3. Semantic endpointing. The model predicts completion from the words said, not just the silence.

4. Fused acoustic-semantic models. "Models fusing acoustic and semantic signals predict completion instead of waiting for silence," which lets them trigger a turn ending before trailing silence has even accumulated - the only real path to beating the silence-timer tax.

The core tradeoff: the Goldilocks problem

Endpointing is a tuning problem between two failure modes: fire too early and you interrupt the user mid-sentence; fire too late and the agent feels dead. There is no single right silence timeout.

The subtler point for ML leads is that consistency beats average. A model with a great median but a long p95/p99 tail produces the occasional two-second hang that users remember and churn over. Watch for configurations where a hard wait timer silently overrides your semantic signal - you may have paid for a predictive model and then capped it behind a fixed timeout.

Configuration in practice

The OpenAI Realtime API makes the two philosophies explicit. "VAD has two modes: server_vad: Automatically chunks the audio based on periods of silence. semantic_vad: Chunks the audio when the model believes based on the words said by the user that they have completed their utterance."

Choose by use case. For predictable, short command-style turns, silence-based server_vad with a tight timer is often fine. For open-ended conversation where users pause mid-thought, semantic detection avoids interruptions at the cost of a little more decision time. If you're weighing hosted realtime APIs against running the stack yourself, we cover the architecture tradeoffs in our write-up of an alternative to the OpenAI Realtime API.

Measuring what you're optimizing

Treat endpointing latency as a first-class metric, not a config detail buried in a VAD library. There are really three latencies to track separately:

1. Endpointing latency - the dead air between the user's last word and the turn decision.

2. Processing latency - STT finalization + LLM + TTS.

3. Network latency - the hops between services, which you can collapse by co-locating components; see our note on network latency in the inference pipeline.

The headline number to report is Time to First Audio. "The most important latency metric for conversational AI systems is Time to First Audio (TTFA) - how long it takes for the agent to start speaking after the customer finishes." Report it at p50, p95 and p99 - the tail is where perceived quality lives.

Measurement only matters against a realistic end-to-end baseline. On Cerebrium's own serverless infrastructure we measured a Pipecat/Ultravox/Cartesia pipeline at 600ms first-time-to-audio, and a co-located transcription-plus-agent stack at roughly 500ms end-to-end. The lesson from our benchmarks is architectural: in a RAG voice agent we saw an external LLM API cost ~800ms versus ~100ms for a locally hosted model, as detailed in our realtime RAG voice agent build. Where your components run shapes how much of the ~800ms budget endpointing is allowed to keep.

For the pieces downstream of the turn decision, we go deeper in dedicated guides on streaming STT latency, time to first audio for TTS, and the broader speech-to-speech versus cascaded latency question. Deployment and scaling details live in our Pipecat voice agent GPU deployment guide.

Key takeaways for ML leads

• Endpointing is the front-of-chain bottleneck: its latency is added to every turn, so profile it first.

• Keep the terms straight - VAD is frame-level, endpointing is turn-level, and semantic/fused models are what let you escape the silence-timer tax.

• Budget against ~800ms to TTFA and treat endpointing latency as a named metric reported at p50/p95/p99.

• Consistency beats a good average; the p99 hang is what users churn over.

• Architecture sets your ceiling - co-location and local model hosting buy back the budget endpointing needs.