Real-Time Video Generation Latency on Serverless GPU

Real-time video generation on serverless GPU is feasible only when per-frame inference plus the serverless tax fits a hard budget: 33.3ms at 30fps, 16.7ms at 60fps, with p99 under 100ms. If your model can produce a frame inside that window and your platform can keep a worker warm, serverless works. If either condition fails, you drop frames. This guide walks a Lead ML Engineer through the budget, where it goes, and a go/no-go framework for serving generated avatar frames inside a voice-AI agent.

The per-frame latency budget: what "real-time" requires

Real-time video means every frame is ready before the next one is due. At 30 FPS that is 33.3 milliseconds per frame end to end, covering capture, preprocessing, inference, and postprocessing; at 60 FPS the window halves to 16.7ms. The industry framing is explicit: real-time inference "returns each prediction before the next input arrives, such as within 33 ms per frame at 30 FPS."

A single average number is not enough. For live stream analysis, cloud deployments target a tail SLA of p99 < 100ms per frame so that occasional slow frames do not stall the stream. Treat 33.3ms (or 16.7ms) as your median target and p99<100ms as the ceiling you must not blow past.

Where the budget goes: glass-to-glass vs GPU-only

The number you benchmark on the GPU is not the number your user feels. Glass-to-glass latency includes network transport, decode, preprocessing, inference, postprocessing, encode, and playback. GPU-only inference is one slice. A frame that renders in 17ms on the device can still miss its deadline once you add transport and queueing, which is why network latency across the inference pipeline has to be budgeted separately from model time.

Voice-AI pipelines make this concrete. In our own LiveKit -> Cerebrium -> Cartesia commentator build, the model takes roughly ~500ms to interpret frames and Cartesia takes ~180ms to generate voice; the full breakdown is in our realtime AI commentator walkthrough. That is a per-turn budget, not a per-frame one, but it shows how each hop consumes the envelope you have left for rendering frames.

Serverless cold starts: the hidden tax

The single biggest risk to a serverless frame budget is the cold start. An unoptimized serverless GPU worker can spend tens of seconds pulling a container and loading model weights before it serves a single frame, which obliterates any 33.3ms target. We covered the full anatomy of this in Serverless GPU Cold Starts: Killing Voice AI Latency and will not repeat it here.

The fix is pre-warmed worker pools. Optimized platforms advertise sub-200ms cold starts by keeping workers ready before requests arrive; one vendor states its cold starts "are under 200ms for most workloads." Cerebrium's FlashBoot-style pre-warming exists for exactly this. Once a worker is warm, the cold start no longer touches your per-frame path, and the only startup cost that remains is the model's own first-frame render. For comparison, state-of-the-art streaming models render the first frame within ~0.5s. Keep workers warm with reserved GPU capacity if your traffic cannot tolerate even an occasional cold worker.

The real bottleneck is memory bandwidth, not FLOPs

For streaming diffusion and transformer video models, per-frame latency is bound by how fast you can move weights and activations through the GPU, not by raw compute. Memory bandwidth is the lever. The H100 delivers 3.35 TB/s; the H200 raises that to 4.8 TB/s, a 1.4x improvement that "indicates a capability to feed data to the processing cores much faster," worth 42.4% higher offline throughput on Llama 70B. The B200 pushes to 8.0 TB/s with 192GB HBM3e, roughly 2.3x the H100, which also lets larger models fit on a single GPU and cuts multi-GPU communication overhead.

The practical read: choosing a higher-bandwidth GPU often buys more frame-rate headroom than any software trick. See our H100 vs H200 inference throughput and B200 throughput and TTFT breakdowns before you size hardware.

What hits real-time today

State-of-the-art streaming video generation already clears 30 FPS on the right hardware. StreamDiffusionV2 renders the first frame within 0.5s and attains 58.28 FPS with a 14B-parameter model on four H100 GPUs, without TensorRT or quantization. That is roughly 17ms per frame, comfortably inside the 30fps budget and close to the 60fps line. The catch is the cluster: those numbers come from four H100s, not a single serverless replica, so your cost-per-frame and multi-node serverless cluster topology matter as much as the model.

For batch or near-real-time work where you can tolerate higher latency, text-to-video and image-to-video pipelines run fine on single GPUs; if that is your regime, compare approaches in our Sora API alternatives and fal.ai alternatives for image/video generation guides.

Decision framework: can serverless GPU meet your frame budget?

Work through these in order:

1. Fix the budget. 33.3ms/frame for 30fps, 16.7ms for 60fps, p99<100ms. Decide which you truly need; avatars for a voice agent often read well at 24-30fps.

2. Measure GPU-only per-frame time on your target hardware. If it already exceeds the budget, no serverless optimization saves you, upgrade the GPU or shrink the model.

3. Subtract the serverless tax. Warm workers must be the norm. Unoptimized cold starts of tens of seconds are disqualifying; sub-200ms pre-warmed starts plus a ~0.5s first frame are acceptable.

4. Add transport. Glass-to-glass must still clear p99<100ms after network and encode.

5. Check headroom with bandwidth. If you are 5-10ms over, a jump from H100 (3.35 TB/s) to H200 (4.8 TB/s) or B200 (8.0 TB/s) may close the gap.

Go if warm GPU-only time leaves margin under your budget and you can keep workers warm. No-go if you need 60fps from a single replica on a model that only hits 17ms on a four-GPU cluster. In our testing of voice-AI stacks on Cerebrium, keeping the model local on serverless infrastructure rather than calling an external API is what makes the latency math work: an external LLM API adds ~800ms versus ~100ms locally, a pattern documented in our realtime RAG voice agent build. The same principle governs frame serving, co-locate the model with the pipeline and keep the worker warm.

If your real workload is a talking-head avatar driven by speech, pair this frame budget with a tight voice stack; we hit 600ms end-to-end first-time-to-audio in our Ultravox deployment.