vLLM Replica Startup Time: Compile Cache & GPU Snapshots

When a DevOps engineer scales a vLLM deployment, the question is rarely "is the model fast?" It is "how long until a newly added replica actually serves traffic?" That gap, the vLLM replica startup time, decides whether autoscaling absorbs a burst or drops requests while a fresh pod grinds through initialization. The good news: a cold start is not one monolithic delay. It is a sequence of phases, and compile cache and GPU memory snapshots each remove a different one. Measure the phases, then pick the right lever.

Anatomy of a vLLM replica cold start

A vLLM worker does not become ready the moment the container runs. It moves through several preparation steps. vLLM's own reconfiguration flow describes joining workers needing to "complete the required compilation, kernel warmup, tuning setup, and CUDA graph construction." Add weight loading at the front, and the practical phase breakdown is:

• Weight loading - pulling the checkpoint from storage into GPU memory. For a Llama-3-8B this alone runs tens of seconds; we cover the storage side in loading model weights from object storage.

• Compilation - Python bytecode and Torch compilation of the model graph.

• CUDA-graph capture - recording graphs for low-overhead replay.

• Kernel warmup and tuning - autotuning and first-run kernel JIT.

Each phase has a different fix. Lumping them into one "cold start" number hides which one you should attack first.

Why this is a scaling problem, not a one-time cost

If startup were paid once at deploy, you could ignore it. It is not. Every replica your autoscaler adds pays the full phase sequence again. Under bursty traffic, that cost lands exactly when you can least afford it, at the moment demand spikes and you are spinning up capacity. This is why replica startup is a throughput-and-SLO concern, not a deployment footnote. We walk through the autoscaling math in autoscaling a Llama-70B inference replica. The slower each replica, the more idle buffer capacity you must keep hot to hide the latency, which directly inflates your LLM inference cost at scale.

Phase fix: a persistent compile cache

Compilation is the phase a compile cache removes. Shipping precompiled Python bytecode in the container image is one of the cleanest wins available. In vLLM's H100 benchmarks, shipping bytecode in the image saved "about 18 seconds per startup" with a single UV build setting. The effect held across model sizes: for Qwen3-8B and Qwen3-32B, "median paired startup savings were 17.6 and 17.8 seconds, respectively." Even on commodity hardware the pattern repeats; on an RTX 4090 laptop across several small models, "all 18 pairs favored precompilation, with median paired savings of 12.8–13.5 seconds."

The lesson for DevOps: bake the compile cache into your image rather than regenerating it on every boot. It is deterministic, portable, and costs nothing at runtime. Combine it with tuned engine flags, covered in vLLM server parameters for inference throughput, so the server that starts faster also serves faster.

Phase fix: GPU memory snapshots

Compile cache attacks compilation. GPU memory snapshots attack everything downstream of it, including the expensive weight-load phase. A snapshot checkpoints a running CUDA process, model weights already resident in GPU memory, warmed kernels and captured graphs, then restores that image instead of cold-booting the container. Infrastructure teams treat snapshots as a core elasticity primitive alongside "process checkpointing and GPU memory snapshots" to "reduce the work needed to bring another model replica online."

The payoff is large. When we measured snapshots across a benchmark of six workloads, 100 cold-start requests each over 24 hours on matched GPU classes, "Cerebrium snapshots reduced cold starts by an average of 71% compared to running the same workloads on Cerebrium without snapshots, with reductions as high as 88% on vLLM." That is the full method and curve; we also reworked the layer underneath, cutting AWS node boot time by 83%. Snapshots are the reason a serverless platform can advertise cold start times of 2 to 4 seconds where unoptimized stacks measure far longer.

Decision table: compile cache vs GPU snapshots

The two are complementary, not competing. Compile cache shrinks the compilation phase deterministically inside your build; snapshots collapse the remaining phases at the platform layer. Use both. For a broader provider view, see Modal alternatives for serverless GPU inference.

DevOps checklist: measuring and verifying replica startup

1. Instrument per-phase timing. Log timestamps at container start, weights-loaded, compilation-done, graphs-captured, first-token-served. One aggregate number tells you nothing about which lever to pull.

2. Confirm the compile cache is actually warm. Verify bytecode ships in the image and is not regenerated per boot.

3. Measure a real cold start, not a warm one. Force a fresh replica and capture the whole sequence, as we did measuring cold-start reductions on vLLM.

4. Benchmark snapshot restore against cold boot on your own model and GPU class.

5. Track the tail, not the median. The slow replica is the one that breaks your SLO during a burst.

Key takeaways

vLLM replica startup is a stack of phases, and each has a dedicated fix. A persistent compile cache deterministically removes the compilation phase, worth roughly 18 seconds on H100 in vLLM's benchmarks. GPU memory snapshots collapse weight load, warmup and graph capture, worth up to 88% on vLLM in Cerebrium's own testing. Measure per phase, bake the cache into your image, and let snapshots handle the rest.