S2S vs Cascaded Voice Agent Latency: A Buyer's Guide

TL;DR / the decision box

If you are choosing between an end-to-end speech-to-speech (S2S) model and a cascaded STT → LLM → TTS pipeline for a voice agent, here is the short version:

• S2S wins on structural latency. It removes two model hops and the text conversions at their boundaries, and it preserves non-linguistic signal (tone, hesitation, overlap) that text throws away.

• Cascade wins on swappability. Every stage is an independent, replaceable component. When a provider deprecates a model, you swap one box instead of re-architecting the whole agent.

• Latency is a budget, not a verdict. Both architectures can land inside a conversational window. The real question is which stage you will have to replace, and whether your architecture lets you replace it without a rebuild.

This guide reframes the comparison away from "which is faster" and toward "where does latency come from, which stage will you swap, and does your design let you swap it?"

The two architectures

A cascaded voice agent runs three models in sequence. As one external write on the topic puts it, "speech-to-text transcribes the other side, an LLM writes a reply, and text-to-speech reads it out." Text is the interface at every boundary. That is both the strength and the weakness: the LLM in the middle is a normal text model that is good at tools, and every piece is swappable - but turning speech into text throws away everything that isn't words.

An S2S model collapses that into one model on the audio path. The same source describes it bluntly: "There's no speech-to-text step, no text-to-speech step, and no resampling. Every stage you skip is latency you don't pay and a failure mode you don't have." Audio goes in, audio comes out, and non-linguistic information survives the trip.

That is a genuine concession worth making up front: S2S has a structural advantage. We are not here to claim the cascade always wins.

Debunking "the cascade is always slower"

The common shorthand - cascade = three hops = slow, S2S = one model = fast - is too coarse to plan with. Latency in a voice agent is not the arithmetic sum of three worst cases. It is a set of overlapping component budgets that can be streamed and pipelined: STT emits partials while the user is still talking, the LLM begins decoding before STT finalizes, and TTS starts speaking the first words before the LLM has finished the sentence.

Conversely, "end-to-end" does not mean "automatically fast." An S2S model still has its own time-to-first-audio, and if it is reached over a slow network or a cold GPU, its single-model simplicity buys you nothing. The architecture sets a floor; your deployment sets the number you actually ship. We cover the deployment half of that in our writeup on building a global-scale voice agent inside a 500ms budget.

Component-level latency budget

The only honest way to compare is per stage, with real numbers. These are Cerebrium's own measurements from production voice-agent builds:

• STT. Deepgram's time-to-first-byte is ~250ms through the hosted API versus ~110ms when the model runs co-located on Cerebrium - a 140ms saving, purely from removing the network hop. The method is in our multilingual transcription guide and our Faster-Whisper tuning writeup.

• LLM. Calling an external LLM API costs roughly ~800ms versus ~100ms for a model hosted locally - the single largest swing in the stack. The full pattern is in our realtime RAG voice agent build.

• TTS. A locally deployed TTS model returns first audio in ~80ms versus ~150ms through a hosted API, a ~90ms saving. See our Orpheus-at-scale and Sesame CSM deployments.

In a separate real-time build, we measured ~500ms for the model to interpret frames plus ~180ms for Cartesia to generate the voice frames - a reminder that in a cascade, where each component runs dominates the total. Co-locating the stack is how teams reach roughly ~500ms end-to-end latency. The takeaway: most of the cascade's latency is not the hops themselves, it is the network between hops. Remove the network and the structural gap to S2S narrows sharply.

Beyond latency: what each architecture costs you

Latency is one axis. The trade matrix has two more:

• Cascade buys control and inspectability. You can log the transcript, inject tool calls, redact PII at the text boundary, and route the LLM independently. Each stage is observable and testable. The cost is a latency floor set by the boundaries and the loss of non-linguistic signal.

• S2S buys a lower floor and expressiveness. It preserves tone and timing and removes conversions. The cost is that the model is a black box on the audio path - harder to inspect, harder to patch one stage of, and tied to whoever ships the single model.

Neither is strictly better. The choice depends on which property your product cannot live without.

The durability argument: a deprecation worked example

Here is why we weight swappability so heavily for commercial agents. In October 2026, OpenAI emailed customers that three TTS models - tts-1, tts-hd, and gpt-4o-mini-tts - will be shut off on January 6, 2027. As one affected user reported, the shutdown email "points to the deprecations page for recommended replacements, but as of October 1, 2026 that page has no entry for any of these models."

Sit with what that means for each architecture:

• In a cascade, a TTS deprecation is a one-box swap. You replace the TTS stage with another model - open or hosted - revalidate one component's latency and voice quality, and ship. Your STT and LLM never move.

• In an S2S deployment, the voice is welded to the model on the audio path. If the single model is deprecated or its voice changes, you re-architect and re-test the whole agent at once.

Latency risk is not only milliseconds today; it is the milliseconds you are forced to re-budget when a dependency disappears. Swappability is a hedge against that risk. We expand the TTS-replacement angle in our guide to OpenAI TTS API alternatives and time-to-first-audio.

The open TTS and evaluation landscape

The hedge only works if real replacements exist. They do: open TTS models such as Orpheus- and Sesame-class voices can be self-hosted to hit the ~80ms TTFB figures above, which is exactly what makes the cascade's one-box swap practical rather than theoretical. The same logic applies to STT and the LLM: each stage has a competitive, swappable field. For the LLM leg specifically, co-locating an open model is what converts that ~800ms API call into a ~100ms local one. If you are building the realtime transport around this, our alternative to the OpenAI Realtime API and Ultravox deployment are the two reference shapes.

Decision checklist

Choose S2S when: expressiveness and the lowest possible floor matter more than inspectability, your conversation design leans on tone and timing, and you can tolerate re-testing the whole agent if the model changes.

Choose a cascade when: you need transcripts, tool calls, and stage-level control; you want to swap any one provider under deprecation without a rebuild; and you can co-locate the stack to recover most of the structural latency gap.

For either, size the GPU side early - our notes on concurrent voice sessions per GPU and voice-AI cold starts matter as much as the architecture choice.

How to benchmark on your own traffic

Do not decide on a vendor slide. Measure:

1. Instrument per stage. Record TTFB for STT, LLM first token, and TTS first audio separately, plus the network time between them. For S2S, record time-to-first-audio end to end.

2. Replay real calls, not clean studio audio - include overlap, hold music, and accents.

3. Measure at your p95, not your median, and under concurrency, because the floor you advertise must hold when every session is live.

4. Run the swap drill. Replace one stage and re-measure. The time that drill takes is your durability number.

Latency is a budget. Spend it where it buys the most conversation, and architect so you can re-budget the one stage you will inevitably have to replace.