Pipecat Voice Agents: Which Stage Belongs on a GPU
Connor Blier
Founding GTM
Pipecat Voice Agents: Which Stage Belongs on a GPU
A Pipecat pipeline is four services in a row, and only three of them are worth moving onto your own GPU. In our testing the speech-to-text hop saves 140ms, the language model hop saves 400ms or more, and text-to-speech saves 70ms. Co-locating them saves another 150ms on network alone.
Pipecat gives you a pipeline: transport in, speech to text, a language model, text to speech, transport out. The framework does not care where each service runs, which is exactly the decision that decides whether your agent answers in 600ms or in two seconds. The pipecat voice agent gpu deployment question is really four smaller ones, one per stage, and each of the four has a measurable answer.
Start from the only number the caller experiences. Everything below is a component of time-to-first-audio, and a Pipecat pipeline is only as fast as the sum of its hops plus the network between them. In our testing, three of the four stages move that number and one does not.
Which Pipecat services actually benefit from a GPU?
Three of the four. Transport is I/O and belongs wherever your users are. The other three are model inference, and each one measures differently.
| Pipecat stage | Hosted API | On our GPUs | Delta |
|---|---|---|---|
| Speech to text (Deepgram) | ~250ms TTFB | ~110ms TTFB | 140ms |
| Language model | 700ms to 1.5s TTFT | ~300ms TTFT | 400ms to 1.2s |
| Text to speech | ~150ms TTFB | ~80ms TTFB | ~70ms |
| Network between stages | public internet | ~2ms inter-cluster | 150ms+ |
The speech-to-text row comes from running Deepgram's model ourselves: the TTFB from the API is ~250ms against ~110ms hosted locally. The language-model row is the widest and the most under-appreciated, because it is not just slower but variable: OpenAI 4o-mini varies from 700ms to 1.5s while a self-hosted llama-3-8b holds ~300ms consistently. A voice agent budget is set by the worst case, not the median, so a stage with a 800ms spread costs you the whole spread. The text-to-speech row is the smallest single win, ~80ms against ~150ms, and it is corroborated by a completely different model: our Orpheus deployment measured a TTFB of ~100ms.
The row most people leave out
The fourth row is not a model at all. When these services sit in different clouds, every Pipecat frame crosses the public internet twice. Running them beside each other, inter-cluster routing costs ~2ms and saves 150ms or more on its own. That is larger than the text-to-speech saving and it requires no model change whatsoever. The routing layer that makes it possible is Thalamus, our own request router.
Add the rows up and the budget lands where you would expect. A different pipeline shape, Ultravox paired with Cartesia, reached 600ms end to end, first time to audio on the same infrastructure. The point is not that one vendor stack wins. It is that the stage-level arithmetic is the same for every Pipecat graph you can draw.
What it costs to run
Latency is half the decision. Capacity is the other half, and it is the half that makes the per-minute price computable.
A single A10 handles roughly 160 to 180 concurrent speech-to-text connections, at $1.44 per hour per instance. Text to speech sits on an A10 at $1.2 per hour for 180 concurrent conversations. Divide the hourly rate by the concurrency it sustains and the whole pipeline resolves to $0.02932 per minute per call. We walk that arithmetic out in full in our per-minute voice agent cost breakdown.
Two things follow from that number. First, per-call economics are decided by concurrency per GPU, not by the GPU's hourly rate, so benchmark concurrency on your own audio before you model a price. Second, self-hosting only pays below a utilisation floor: at low volume an API is cheaper, and the crossover is where your fleet stays busy. Our alternative to the OpenAI Realtime API covers that trade in more detail.
Where the remaining latency hides
Once the three model stages are co-located you are usually left fighting two things.
Tool calls are the first. A Pipecat agent that reaches for a function mid-turn pays a second language-model round trip, and our real-time architecture write-up covers how to hide that behind a filler phrase rather than eliminate it.
Non-English audio is the second. Word error rate, not latency, is what breaks multilingual agents, and a retry after a bad transcription costs an entire turn. That failure mode is documented in our multilingual transcription notes. If you are choosing a voice model for quality rather than speed, our Sesame CSM deployment is the other end of that trade.
Current GPU rates are on the pricing page; the figures quoted here come from our published benchmarks and will drift, so re-derive before you budget against them.
The short version
In a Pipecat pipeline, move speech to text, the language model and text to speech onto GPUs you control, and put them next to each other. On our numbers that is 140ms, 400ms or more, and 70ms of model latency, plus 150ms or more of network. Then size the fleet from concurrency per GPU, because that is what sets the per-minute price.
Frequently asked questions
- Does Pipecat require a GPU?
- No. Pipecat is an orchestration framework and runs on CPU. The GPU decision applies to the model services inside the pipeline: speech to text, the language model and text to speech. Transport does not benefit.
- Which Pipecat stage gives the biggest latency win when self-hosted?
- The language model. In our testing a hosted 4o-mini varied from 700ms to 1.5s time-to-first-token while a self-hosted llama-3-8b held around 300ms consistently. The consistency matters as much as the average, because a voice budget is set by the worst case.
- How many concurrent calls does one GPU handle?
- For speech to text, roughly 160 to 180 concurrent connections on a single A10 in our measurements. Text to speech reached 180 concurrent conversations on the same class of GPU. Benchmark your own audio, since concurrency varies with model and sample rate.
- What does a self-hosted Pipecat pipeline cost per minute?
- Our worked example totals $0.02932 per minute per call across speech to text, the language model and text to speech. That figure is a function of concurrency per GPU rather than the GPU's hourly rate.
- Is co-locating the services worth it if I keep using hosted models?
- Partly. Co-location removes the network hops between stages, which measured 150ms or more of saving on its own. But if the models themselves stay behind third-party APIs you keep their variability, which is the larger cost.
Sources
- cerebrium.ai
“Typically, the TTFB (time-to-first-byte) from the Deepgram API is ~250ms whereas the TTFB hosting it locally on Cerebrium is ~110ms which is a 140ms saving.”
The STT stage of a Pipecat pipeline, measured both ways: hosted API against the same model running on our own GPUs.
- cerebrium.ai
“The TTFT (time-to-first-token) of OpenAI 4o-mini varies from 700ms - 1.5s whereas if you deploy a llama-3-8b model or llama-3-70b, you can consistently achieve ~300ms.”
The LLM stage, where the gap between a hosted API and a self-hosted model is widest and least predictable.
- cerebrium.ai
“By deploying these models locally on Cerebrium, you can achieve ~80ms TTFB as opposed to APIs that typically respond in ~150ms, leading to a ~90ms saving.”
The TTS stage, the last hop before audio reaches the caller.
- cerebrium.ai
“By deploying them inside Cerebrium, we take advantage of inter-cluster routing between the services that allows them to incur a “network request latency” of ~2ms, which alone saves us 150ms+.”
What co-locating the stages is actually worth, separately from the per-stage model latency: the network cost of the hops between them.
- cerebrium.ai
“On Cerebrium, a Deepgram service can run on an A10 GPU and can handle roughly 160-180 connections concurrently.”
The capacity figure that turns a per-call latency budget into a fleet size.
- cerebrium.ai
“Therefore the cost for this on Cerebrium is $1.44 per hour per instance and Cerebrium manages all autoscalling.”
The hourly cost of the STT instance in that worked example.
- cerebrium.ai
“They usually require an A10 GPU which will cost $1.2 per hour for 180 concurrent conversations.”
The hourly cost of the TTS instance at the same concurrency.
- cerebrium.ai
“- Total: $0.02932 per minute per call.”
The whole pipeline reduced to one per-minute number, which is the figure a voice product actually has to price against.
- cerebrium.ai
“Using this pipeline, we are able to achieve an end-to-end latency (First time to audio, in just 600 ms).”
An end-to-end result from a different pipeline shape, showing the budget is achievable and not specific to one vendor stack.
- cerebrium.ai
“From our testing this endpoints has a TTFB (Time-to-first-byte) of ~100ms which is perfect for low latency voice applications.”
A second, independent TTS measurement on a different model, corroborating the TTS stage figure above.
