Serverless GPU Cold Starts: Killing Voice AI Latency

The cold-start tax on a single voice turn

When an ML engineer builds a conversational Voice AI agent, the whole experience lives inside a sub-second turn budget. A user speaks, the system runs ASR, an LLM decides and generates, TTS speaks back, and the perceived gap has to feel like a human pause. Serverless GPU cold starts blow straight through that budget.

A serverless GPU platform gives you scale-to-zero billing: when no requests arrive, your GPU shuts down and you pay nothing. The trade-off is that the next request has to wake a cold machine. As one provider puts it, serverless GPU "delivers both, at the cost of a 40-90 second penalty the first time a pod has to load a large model from scratch." For a chatbot that first token can arrive late and be forgiven. For a voice turn, a multi-second gap is a dropped call.

What actually happens during a serverless GPU cold start

Cold start latency is not one delay. It is, per one IaaS provider's definition, "the initial delay encountered when an AI model is loaded and executed after a period of inactivity," caused by "the need to allocate compute resources, load model weights into memory, initialize inference runtimes, and establish any necessary data pipelines." Each of those is a distinct, measurable phase.

Container and image pull

Before any model loads, the platform pulls your container image. LLM inference images are heavy, and the uncached pull dominates everything else: a breakdown of cold-start phases lists "Container pull (uncached) | 4-8 min | 4-8 min | Image size (15-18 GB)." That is minutes, not seconds, driven purely by 15-18 GB of image data crossing the network. Cutting it down is its own discipline, one we dig into in rethinking container image distribution to eliminate cold starts.

Model weight loading

Once the container runs, multi-gigabyte weights move from storage into GPU memory. This scales with model size. The same breakdown shows "Weight load from storage | ~8s | ~40-45s | NVMe bandwidth", roughly 8 seconds for a small model but 40-45 seconds for a 70B-parameter model, bottlenecked on NVMe bandwidth.

CUDA graph warmup

Finally the framework compiles and captures CUDA graphs so kernels run at full speed. Skipping this means slow first inferences; doing it cold costs time. The breakdown lists "CUDA graph cache persistence | Graph capture | 10-30s | Persistent NVMe volume", 10 to 30 seconds spent capturing kernels on every cold boot.

The timeline

Stack the phases and the totals match production reports: "spinning up a GPU container, initializing the ML framework, loading multi-gigabyte model weights into GPU memory, and warming up CUDA graphs can take 30 to 60 seconds." Container pull first, then weights, then warmup: a serial pipeline where every phase pushes back the one after it.

Why cold starts hit real-time Voice AI harder

Most apps tolerate a cold start as a one-time hiccup. Voice AI cannot, because the pipeline is latency-summing: ASR feeds the LLM, the LLM feeds TTS, and the human ear notices every added millisecond. Production serverless LLM deployments "report cold start times exceeding 40 seconds just to produce the first token, while subsequent inference takes only ~30ms per token. This 1000x latency gap between cold and warm states creates an unacceptable user experience for chatbots, voice AI, and any real-time generative application."

To see the target you are protecting, look at warm numbers. In our testing on a low-latency architecture for voice agents, we measure 60-115 tokens/sec and 300-510 ms TTFT on a single NVIDIA B200, with an LLM tool-decision TTFT of 330-510 ms and a perceived time-to-first-audio of ~1.0 s using a filler technique. Cerebrium's own before/after data cut perceived time-to-first-audio from ~5.3 s on a GPT-4o-mini baseline to ~1.0 s. A 40-second cold start does not just degrade that; it obliterates it. Latency-sensitive teams hit the same wall: Tavus scaled human-like AI experiences, bitHuman scaled digital humans 10x faster, and Creatium achieved 18x faster cold starts to stay inside their turn budgets.

Mitigation techniques

The fix is to attack each phase with the technique that recovers its specific latency, and then to remove the repeated startup work entirely.

Warm pools

The surest way to avoid a cold start is to not be cold. Keep a minimum of active workers: "Most platforms let you configure a minimum number of active workers so you always have capacity ready." Requests then land on already-running instances instead of triggering the full pull-load-warmup chain.

Container and image optimization

Because the uncached image pull is the single largest phase, shrinking and caching the image pays back more than any other single change. Slim base images, cached layers, and faster image distribution cut minutes off the front of every cold start; we measured an 83% boot-time improvement on custom container images doing exactly this.

Checkpointing (memory snapshots)

Cerebrium's approach to cold starts is checkpointing, also called memory snapshots. Instead of rebuilding the runtime from scratch on every scale-up, importing libraries, loading model weights, copying them onto the GPU, initializing CUDA, and compiling kernels, Cerebrium snapshots the fully initialized container once it is warm, "including CPU memory, GPU memory, process state, model weights, and compiled kernels," and restores that state directly into a new container. Because both the model weights and the compiled CUDA graphs are already captured in the snapshot, one restore brings back "the same warmed-up runtime we froze earlier: PyTorch has already been imported, model weights are already resident on the GPU, kernels are already compiled, and the application is ready to serve traffic." A single restore removes both the model-loading step and the CUDA-graph warmup step that would otherwise run on every cold start. A checkpoint is created after the first warm boot, so it speeds up subsequent scale-ups rather than the very first startup, and it is opt-in and keyed to the exact GPU type, machine type, and runtime it was captured on.

The payoff is measured, not theoretical. On a g5.12xlarge, a full vLLM cold start of a roughly 9 GB container took around 50 seconds, while restoring from a 9 GiB checkpoint cut startup to 2.25 seconds from S3 and 9 seconds from local NVMe. Across a six-workload benchmark, snapshots reduced cold starts by an average of 71% versus the same workloads without snapshots, with reductions as high as 88% on vLLM. Checkpointing is not the right tool for every workload; if an application already starts in a few seconds, caching may be enough. But where cold starts are dominated by deterministic startup work, it is the single biggest lever.

Decision guide

Putting it together for a Voice AI stack

For a production voice agent, layer the mitigations: keep a warm pool sized to your traffic floor, shrink and cache your container image, and use checkpointing so each scale-up restores an already-warm process instead of rebuilding it. Combined with an efficient serving stack and global request routing, this keeps you inside the sub-second turn budget the whole pipeline depends on, the same discipline behind a real-time RAG voice agent that stays responsive under load.

Key takeaways

• A serverless GPU cold start is a serial stack: container pull, weight load, then CUDA warmup, totalling 30-90 seconds cold.

• Container pull (15-18 GB images) and 70B weight loads (40-45 s) are the biggest single contributors.

• Voice AI's ASR-to-LLM-to-TTS pipeline is latency-summing, so a cold start's 1000x cold-vs-warm gap is unacceptable.

• Match the fix to the phase, then remove the repeated work: warm pools, slim images, and checkpointing, which restores model weights and compiled CUDA graphs in a single step, recover the latency.

Start building on Cerebrium

Ready to keep your own workloads inside the turn budget? Deploy Voice AI and LLM inference on serverless GPUs with fast cold starts, checkpointing, and pay-per-use pricing, no Kubernetes required. Get started for free and have a model in production in minutes.