Real-Time Video Avatar Sessions Per GPU & Sticky Routing

Ask "how many real-time video avatar sessions fit on one GPU?" and most capacity math starts with FLOPs and VRAM. That is the wrong first question. The binding constraint in production is session affinity: whether your router keeps a live avatar session pinned to the GPU that already holds its warm model and conversational state. Get routing right and your sessions-per-GPU number is real. Get it wrong and every mis-routed turn pays a cold start, your effective capacity collapses, and your cost per GPU-hour balloons.

This guide connects routing architecture to the capacity and cost math an ML engineer actually has to defend.

1. The real constraint: session affinity, not just raw GPU compute

A real-time video avatar is a long-lived, stateful session. The model is warm, there is per-session context, and audio/video frames stream continuously. If a mid-session request lands on a different replica than the one holding that warm state, the new replica has to spin up - and you have manufactured a cold start inside an active call.

Sticky routing (session affinity) is the mechanism that prevents this. The router hashes or tags each session to a specific GPU replica and keeps routing that session's traffic there for its lifetime. This is the same class of problem as token-load-aware routing for LLM serving: the router's decision, not the kernel, decides whether a GPU is used efficiently. Sessions-per-GPU is therefore a routing property first and a compute property second.

2. How real-time video avatars run: the native runtime

Modern avatar stacks are separating the transport layer from the execution layer. AlphaAvatar is a clear example. As of v0.6.8, "AlphaAvatar now has its own native realtime Agent runtime, independent of the LiveKit Agents SDK. This does not mean removing LiveKit. LiveKit remains the first supported realtime communication backend. What changes is that LiveKit no longer owns the Agent lifecycle, model execution, tool orchestration, or the application's runtime architecture." The release also "introduces: A native AvatarHost responsible for application and execution lifecycle management."

Why this matters for capacity: once the runtime - not the transport SDK - owns the lifecycle, you own where execution lives. That is exactly the layer where sticky routing is enforced. If you are building the serving side of a pipeline like this, our notes on GPU inference for AI agent workloads and running multiple models on one GPU cover how the components (vision model, TTS, STT) co-reside per replica.

3. The latency contract: TTFB and TTFF

Video avatars live and die on first-frame latency. A session's perceived quality is the time-to-first-byte (TTFB) of audio and the time-to-first-frame (TTFF) of video. These budgets are tight. On the streaming TTS layer we measured roughly 100ms to first byte: "from our testing this endpoints has a TTFB (Time-to-first-byte) of ~100ms which is perfect for low latency voice applications." End to end, in a comparable real-time build, "essentially with every generation we are about 700ms behind the live stream, which is roughly in line with the average commentator."

Sticky routing protects these numbers. A warm, pinned replica answers inside the budget; a mis-routed turn does not. For the full latency breakdown of a cascaded pipeline, see speech-to-speech vs cascaded voice agent latency and network latency in the AI inference pipeline.

4. Cold starts: the cost of a routing miss

A cold start is the penalty for landing on a replica that is not warm. Even on a fast platform the floor is non-trivial: Cerebrium runs "CPU and GPU deployments with extremely low cold start times (2 seconds)." Two seconds is excellent for scaling a new replica to meet demand - and catastrophic if it happens inside a live avatar session because the router failed to keep affinity.

The lesson: cold starts are a scaling tool, not a per-turn event. Keep them at the edge of your capacity curve (adding replicas), never in the middle of a session. We go deep on this in serverless GPU cold starts for voice AI.

5. Sessions and concurrency per GPU: the capacity math

Here is the core of the question. Take a concrete, measured anchor from one component of the pipeline: a TTS model that "usually require[s] an A10 GPU which will cost $1.2 per hour for 180 concurrent conversations." That is 180 concurrent conversations per A10 for that layer - a real sessions-per-GPU figure, not a theoretical one.

To turn that into your avatar number, work per-component:

• Establish the concurrency ceiling of your heaviest per-session model on the target GPU.

• Treat sticky routing as the multiplier that makes the ceiling achievable - without affinity, warm-state reuse breaks and effective concurrency drops toward 1-per-cold-start.

• Size headroom so scaling events add replicas (paying one 2s cold start) rather than evicting live sessions.

The pattern generalises beyond avatars; our companion piece on concurrent voice sessions per GPU walks the same math for pure-audio agents. For a self-hosted vision model, note that "the response times we get with our model are 50% faster and much more reliable since we aren't being rate limited" - reliability per replica is part of what makes a sessions-per-GPU number hold under load.

6. Cost per session: from GPU-hour to per-session economics

Once sticky routing lets you actually hit your concurrency ceiling, cost per session is arithmetic. With "$1.2 per hour for 180 concurrent conversations" on an A10, the per-session-hour cost of that layer is roughly $1.2 / 180 - fractions of a cent per session-hour - but only if the GPU is genuinely running 180 sessions. A routing architecture that keeps replicas half-empty doubles your real cost per session even though the GPU-hour price never changed.

That is the whole argument for affinity-aware capacity planning: sticky routing converts a GPU-hour price into a defensible per-session price. Translate it to a billed unit with voice AI agent cost per minute.

7. Reference architecture and checklist

Putting it together:

1. Own the runtime lifecycle so you control placement (the AvatarHost pattern).

2. Enforce sticky routing - pin each session to its warm replica for the session's life.

3. Protect TTFB/TTFF by answering only from warm, pinned replicas.

4. Keep cold starts at the scaling edge, never per turn.

5. Set sessions-per-GPU from the heaviest component's measured concurrency ceiling.

6. Derive cost per session from the GPU-hour price divided by achieved concurrency, not theoretical.

Do those six and the question stops being "how many sessions fit?" and becomes "what's my cost per session?" - which is the one your finance team actually asks.