Concurrent Voice Sessions Per GPU: The Real Numbers

Ask about concurrent voice sessions per GPU and you will usually get a spec sheet back: memory, FLOPs, tensor cores. None of that answers the question, because a voice agent is not one model on one GPU. It is a pipeline — speech-to-text, an LLM, text-to-speech — and each stage has its own concurrency ceiling, its own latency budget, and its own failure mode when you exceed it.

In our testing the ceiling is set per stage, and the stages are not symmetrical. On Cerebrium, a Deepgram speech-to-text service running on an A10 GPU handles roughly 160 to 180 connections concurrently. Text-to-speech models typically want an A10 as well, and one serves about 180 concurrent conversations at $1.2 per hour. The speech-to-text instance runs $1.44 per hour. Those are the real planning units.

Why "sessions per GPU" is the wrong unit on its own

A concurrency number is only meaningful next to a latency budget. You can push far more sessions onto a GPU than the numbers above if you are willing to let time-to-first-audio drift — and for a voice agent, drifting is the same as failing. Humans notice conversational latency well below a second.

So the honest framing is: how many sessions fit while the pipeline still responds fast enough to feel like a conversation. That reframes the whole exercise, because it means the ceiling depends on where each stage runs, not just how big the GPU is.

Where the latency budget actually goes

This is the part that surprises teams. Co-locating the models with your orchestration buys more than tuning the models does.

For speech-to-text, the time-to-first-byte from the Deepgram API is typically around 250ms, while hosting the same model on Cerebrium gives about 110ms — a 140ms saving for a change that involves no model work at all. Text-to-speech behaves the same way: roughly 80ms when deployed locally against about 150ms through a hosted API, a further ~90ms. We measured a comparable figure independently on Orpheus TTS, at about 100ms time-to-first-byte, which matters because it means the band is a property of co-location rather than of one vendor's model.

The LLM stage is the largest single line. OpenAI's 4o-mini varies from 700ms to 1.5s time-to-first-token, whereas a self-hosted Llama 3 8B or 70B consistently achieves around 300ms. Assembled end to end, we have measured a complete Ultravox and Cartesia pipeline at 600ms first-time-to-audio.

Read that table as a concurrency document rather than a latency one. Every millisecond you win back is headroom you can spend on packing more sessions onto the same GPU before the conversation starts to feel wrong.

Turning concurrency into cost per session

Once you have a concurrency ceiling and an hourly rate, per-session economics are arithmetic. A text-to-speech A10 at $1.2 per hour serving 180 concurrent conversations is two-thirds of a cent per conversation-hour at full packing. The speech-to-text instance at $1.44 per hour across 160–180 connections lands in the same neighbourhood.

The number that matters operationally, though, is per-minute-per-call across the whole pipeline, because that is what scales with your traffic rather than with your fleet. On the global-scale voice architecture we published, the assembled total came to $0.02932 per minute per call. We broke that figure down stage by stage in the voice agent cost-per-minute analysis.

The trap is planning against peak concurrency. Per-session cost is instance cost divided by average concurrency, and on real traffic average sits well below peak. Size for peak and bill for average, and the two numbers diverge quickly.

What changes the ceiling

Four things move these numbers, roughly in order of leverage:

1. Where the stage runs. Co-location is the single biggest lever, as the table above shows, and it requires no model changes.

2. Which model you pick. A smaller speech-to-text or text-to-speech model raises the ceiling directly; multilingual transcription changes the calculation again, because language coverage and model size trade against each other.

3. Where your users are. Regional placement moves round-trip time more than model choice does for anything interactive — which is why we built multi-region deployments and a distributed router for real-time workloads.

4. What else is in the pipeline. Retrieval, tool calls and web search each add a stage. A low-latency architecture with real-time web search and a realtime RAG voice agent show what the budget looks like once those are in.

How to measure this on your own stack

Do not inherit our numbers. Measure the same three things, in this order:

1. Time-to-first-byte per stage, in isolation, before assembling the pipeline. Stage-level regressions are invisible end to end.

2. The concurrency at which time-to-first-byte starts to degrade — not the concurrency at which the GPU runs out of memory. The first is your real ceiling and it arrives earlier.

3. Average concurrency on production traffic, over a full weekly cycle. That is your cost divisor.

If you are weighing this against a managed realtime API, our comparison of self-hosting versus the OpenAI Realtime API covers the trade directly. Current per-second rates for the GPU classes referenced here are on the pricing page — we have not quoted list prices in the body, because they move and a stale price is worse than no price.

The short version

One A10 carries roughly 160–180 concurrent speech-to-text connections, or about 180 concurrent text-to-speech conversations. But the ceiling that binds in production is the latency budget, not the GPU, and co-locating your models buys back more headroom than tuning them does.