Voice Cloning Inference: Cost & Latency Per Request

Every voice cloning decision eventually collapses to two numbers you can put next to any option: how long until the first audio byte arrives (TTFB), and what one synthesis request costs. Providers quote per character, per minute, per credit, or per GPU-hour, which makes direct comparison almost impossible. This guide normalizes all of it so a Lead ML Engineer can decide where a managed API ends and self-hosting begins.

TL;DR - the per-request decision in one table

If you run under roughly 10 million characters a month, or you need voice cloning live tomorrow, a managed API is almost always cheaper once you price your own time. Past that, and at steady utilization, self-hosting flips.

How to measure "per request" - defining the units

TTFB vs total synthesis. For conversational voice the metric that matters is time-to-first-byte, not total synthesis time, because streaming lets the client play the first chunk before generation finishes. As Rime puts it, "Streaming reduces time to first playback because the client can consume the first chunk before Rime finishes generating the complete response." Keep both numbers, but budget against TTFB.

Measure at the tail, not the mean. A P99 is the response time 99% of requests beat - the operationally honest number for anything a human waits on. The human threshold for a conversational pause feeling awkward is only about 200–300ms, so a good average with a bad tail still produces a bad agent.

Converting cost bases to per request. Pick one unit of work (say, a 250-character utterance) and convert everything to it: per-1k-char pricing multiplies directly; per-minute pricing needs an assumed speech rate; per-GPU-hour needs an assumed concurrency. Hold text length, language, and streaming on/off constant across every option or the comparison is meaningless.

Managed API latency per request

On raw TTFB, the streaming-first APIs are now extremely fast. Cartesia Sonic reaches roughly 40ms time-to-first-byte using State Space Models instead of transformers. ElevenLabs Flash v2.5 lands around 75ms. Inworld reports P99 TTFB under 100ms for Realtime TTS-2 and under 25ms for its Flash variant, measured server-side.

Those figures are the model alone. The number your user feels is the whole pipeline - speech-to-text, the LLM turn, then TTS, plus network hops. In our commentator build on LiveKit and Cartesia, we measured roughly 180ms for Cartesia to generate the voice frames on top of ~500ms of model frame interpretation. Where those hops land matters: in our global-scale voice agent, co-locating services for inter-cluster routing gave ~2ms network latency and "alone saves us 150ms+." The fuller treatment of where each millisecond goes is in our write-up on network latency in the inference pipeline and on speech-to-speech vs. cascaded latency.

Managed API cost per request

Cost spans roughly 6x across providers. Cartesia sits around $0.038 per 1,000 characters; ElevenLabs lists $0.04 to $0.08 per 1,000 characters depending on model, and $0.05–$0.10 at the higher tiers. On a per-minute basis, Cartesia managed agents are $0.06 per minute and ElevenLabs voice agents are $0.08 per call minute plus language model and telephony.

Voice-cloning-specific costs. Cloning itself can add a credit multiplier - ElevenLabs' Flash model consumes 0.5–1 credit per character - and some clone tiers gate behind higher plans. Fold that into your per-request unit before comparing.

Worked example. At 10 million characters per month, ElevenLabs charges $1,800 to $3,000 depending on plan tier. That is your managed baseline to beat. For a deeper per-minute decomposition across STT, LLM, and TTS, see our voice AI agent cost per minute breakdown and OpenAI TTS API alternatives by time-to-first-audio.

Self-hosted economics (XTTS-2, Kokoro)

The same 10 million characters that cost $1,800–$3,000 on a managed plan run for roughly $8 in GPU compute on an H100 SXM5 running XTTS-2 at spot rates. That gap is the entire case for building.

Cost drivers. XTTS-2 needs only 3–4GB VRAM per worker at FP16 and supports 17 languages with cross-lingual cloning, so you pack many workers per card - a single RTX 5090 (32GB) handles 20–30 concurrent voice cloning streams. In our own production voice agent the TTS component needed an A10 at $1.2 per hour for 180 concurrent conversations, and the STT instance cost $1.44 per hour; the full pipeline summed to $0.02932 per minute per call. How many sessions you actually fit per card is the lever - see concurrent voice sessions per GPU.

Latency vs APIs. XTTS-2 is autoregressive, so its TTFA scales roughly as embedding_lookup_time + (char_count / chars_per_second_on_GPU) - first-byte latency grows with passage length. F5-TTS uses flow-matching diffusion, giving more consistent TTFA regardless of length.

Hidden costs. If you recompute the speaker embedding per request you add 50–200ms of GPU time for no benefit - cache it. Cold starts are the other tax; mitigation is covered in serverless GPU cold starts for voice AI, and deployment patterns in Orpheus TTS at scale and Pipecat voice agent GPU deployment.

Build vs buy - the break-even

The crossover is volume-driven. Against a $30-per-million-character API, self-hosted open-source TTS breaks even at around 49 million characters per month on GPU costing ~$2/hour. Below your break-even, the managed API is cheaper after you price engineering time, on-call, and idle GPU. Above it, and only at steady utilization, self-hosting wins - the $8-vs-$1,800 gap at 10M characters assumes a card you keep busy.

The honest decision: buy while volume is spiky or small; build when volume is large, predictable, and your team already operates GPUs.

Recommendations for the Lead ML Engineer

1. Normalize to two numbers - P99 TTFB and cost per 250-char request - before you read any pricing page.

2. Under ~10M chars/month or pre-product-market-fit: use a managed API (Cartesia for latency, ElevenLabs for voice breadth).

3. Above your measured break-even and with steady load: self-host XTTS-2 (autoregressive, length-sensitive TTFA) or a diffusion model like F5-TTS for flat TTFA; cache speaker embeddings.

4. Budget the whole pipeline, not the model - co-locate STT/LLM/TTS to recover 150ms+ of network latency.

FAQ

See below.