Prefix Caching for Multi-Turn LLM Agents

Multi-turn agents spend most of their compute re-reading their own context. A ReAct agent making ten tool calls might emit 500 output tokens while consuming 800,000 input tokens, because every call re-sends the full system prompt, tool schemas, and conversation history. Prefix caching recovers that wasted prefill, and at a realistic 90% hit rate it cuts effective prefill cost by roughly 83%. The catch on serverless is that the cache does not survive a cold start or a new replica, so your hit rate has to be engineered, not assumed.

If you are sizing agent infrastructure more broadly, our GPU inference for AI agent workloads guide covers model sizing and routing; this piece stays focused on the prefill/KV-cache layer.

Why prefill, not generation, is the bottleneck

Production agents are not compute-bound on generation. They are compute-bound on context. The Spheron context-engineering write-up puts the input:output ratio for a ten-call ReAct session at 100:1, and notes that "the model generation step takes milliseconds. The prefill pass over 80,000 input tokens takes seconds, and you pay for it on every call."

That prefill is expensive in absolute terms. Generating the KV cache for a 72K-token input on an A100 takes about 30 seconds. Every turn that re-prefills that context is paying the full tax again.

What prefix caching is and what it reuses

KV reuse

Prefix caching stores the KV cache of prior requests and lets a new request skip the compute for any prefix it shares. vLLM's Automatic Prefix Caching "caches the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix." For agents this is a near-perfect fit: system prompts and tool schemas are fixed, so as the KVFlow paper observes, "these prompts remain constant across iterations" and caching "avoids redundant computation on static content."

vLLM vs SGLang implementations

The two dominant engines match prefixes differently. vLLM hashes 16-token blocks; SGLang's RadixAttention uses a radix tree, which reaches up to 6.4x higher throughput on prefix-heavy traffic. At 80% shared prefix and 50 concurrent requests, TTFT p50 drops from 310 ms on vLLM to 195 ms on SGLang, a 37% reduction. We compared both engines end to end in our vLLM, SGLang and TensorRT benchmark.

What it recovers: cost and TTFT

Cost math

Because prefill dominates, cache hits translate almost directly into cost cuts. At a 90% hit rate against a 92% prefill fraction, the saving is 0.90 x 0.92 = 82.8%. This is not theoretical: OpenAI, Anthropic and DeepSeek have all integrated prefix caching and report lowering inference costs by over 50%. For the full unit-economics picture, see our LLM inference cost at scale guide.

Latency math

The same reuse collapses time-to-first-token, since TTFT is dominated by prefill. The 310 ms to 195 ms drop above is the reuse effect at 80% overlap. For context on where TTFT sits in a full request, in our benchmarks vLLM hit the lowest TTFT of 123 ms on 1xH100 at 256 input tokens, batch size 1, measured as the total roundtrip to the Cerebrium platform.

Hit rate decides both

Everything above scales with one number. As the llm-d team puts it, "the KV-cache hit rate is the single most important metric for a production-stage AI agent. It directly affects both latency and cost." Character.AI runs KV cache reuse at 95% while serving over 20,000 queries per second, which is the target to aim for.

The serverless catch

Scale-to-zero destroys the cache

The KV cache lives in GPU memory. When a serverless replica scales to zero, that memory is gone. Worse, the first request after a cold start pays a load penalty: Spheron measures a 40-90 second penalty "the first time a pod has to load a large model from scratch," and the cache does not survive that restart. So your first post-idle turn gets zero cache benefit and the full prefill.

Multi-replica breaks single-instance caching

APC is a single-instance optimization. As llm-d notes, "the moment you scale to a distributed, multi-replica environment, these finely tuned optimizations can fall apart" because requests scatter across pods and a follow-up turn can land on a replica that never saw the prefix. Naive round-robin routing turns a 95% hit rate into a coin flip.

The cold-start floor

The good news is the floor can be low with the right engine packaging. On Cerebrium, the TensorRT-LLM Llama 3 8B engine takes roughly 10-15 seconds to load the model into GPU memory on a cold start, well under the 40-90 second range for naive large-model loads. We cover the mechanics of driving this down in serverless GPU cold starts for voice AI, and how platforms differ in Modal alternatives for serverless GPU inference.

Engineering hit rate on serverless

Three levers matter:

1. Keep replicas warm. Scale-to-zero is the enemy of cache hit rate. Holding a minimum warm replica preserves the KV cache across turns and eliminates the cold-start prefill on the next request.

2. Route to the instance holding the prefix. In multi-replica deployments, session-aware routing that pins a conversation to the replica that already cached its prefix is what keeps hit rate near the Character.AI 95% figure rather than collapsing it.

3. Accept a recompute tax where it is cheaper. When neither is possible, budget for the full prefill and keep the prefill itself fast.

Throughput headroom makes that tax survivable. SGLang reached 460 tokens per second at batch size 64 in our benchmarks, and the TensorRT-LLM tutorial achieves ~1700 output tokens per second (FP8) on a single A10. Once you are pushing tokens across a network, prefill savings can be swamped by round trips, so also tune the path itself, see network latency in the AI inference pipeline and the TensorRT-LLM on serverless GPUs walkthrough. Our global voice agent architecture shows the warm-replica pattern in a latency-critical setting.

Key takeaways

• Agent traffic is mostly re-sent context; prefill, not generation, drives cost and TTFT.

• At a 90% hit rate you recover ~83% of prefill cost, and TTFT drops sharply on shared prefixes.

• Hit rate is the single metric that governs both, and on serverless it is not free: scale-to-zero and multi-replica scale-out both destroy the cache.

• Engineer hit rate with warm replicas and session-aware routing, and keep cold starts and prefill fast so the recompute tax stays cheap.