Serverless vs Always-On GPU: LLM Break-Even

The short answer

Serverless GPUs bill you only while a worker is running, so they win when your traffic is bursty and your hardware would otherwise sit idle. Always-on GPUs bill continuously, so they win once you keep the accelerator busy enough to amortize the idle hours. On a RunPod A40, the crossover sits at roughly 29 minutes of billed time per hour (about 48% utilization) - below it serverless is cheaper, above it an always-on pod is. When both are fully saturated, the same benchmark found a busy serverless worker costs about 2.3x more per token than a pod on identical hardware.

That premium is the price of elasticity. The question for a CTO is not "which is cheaper" in the abstract - it is "what is my steady-state utilization, and does it sit above or below the break-even line."

The break-even formula

Every comparison reduces to cost per 1M tokens:

$ per 1M tokens = hourly GPU price / (tokens per second x 3600) x 1,000,000

The numerator is what you pay; the denominator is what you produce. For an always-on pod the hourly price is fixed whether or not requests arrive, so your effective cost per token is only as low as your busy utilization allows. For serverless, you pay the (higher) per-second rate but only for billed utilization - the fraction of the hour a worker is actually spun up.

Those two words matter. Busy utilization is how hard the GPU works while it is on. Billed utilization is how much of the wall-clock hour you are charged for. Serverless decouples them: scale-to-zero means billed time can be far below the full hour. An always-on pod collapses them into one - you are billed for all 60 minutes regardless. We walk through the idle-billing side of this trade-off in more depth in reserved GPU capacity for production LLM inference.

A concrete break-even: the RunPod A40

RunPod's own A40 benchmark gives a clean worked example. A busy serverless worker cost about 2.3x as much per token as a pod on the same GPU when both were fully utilized. Working the economics backwards, serverless only comes out ahead when its flex workers would be billed for under about 29 minutes of each hour - roughly 48% utilization.

So the decision rule is blunt: if a single GPU's worth of your traffic keeps that GPU busy more than half the hour, rent it always-on. If your traffic is spikier than that - a few busy minutes, then quiet - serverless scale-to-zero wins despite the per-token premium. For a larger-model version of this same crossover, see Llama 405B serverless GPU inference cost.

GPU hourly rates to plug in

The numerator is just a number you look up. Recent on-demand rates (Runcrate, October 2026):

A cheaper card is not automatically the cheaper token. The H100 costs nearly 5x the A6000 per hour, but if it produces more than 5x the tokens per second for your model, it wins on cost per token. That is why the denominator - throughput - usually decides, and throughput is dominated by concurrency.

The concurrency cost curve

Here is the part that quietly moves the break-even line. The same hardware gets dramatically cheaper per token as you batch more concurrent requests onto it. From a vLLM benchmark of Qwen2.5-7B-Instruct on the RTX A6000:

Serving one request at a time, the A6000 managed 42 tokens/sec at $4.86 per 1M tokens. At concurrency 8 it hit 307 tokens/sec and $0.67. At concurrency 32 it reached 1,012 tokens/sec and just $0.20 - a 24x drop in per-token cost from the same GPU, purely from batching.

This reshapes the break-even. High concurrency lowers your always-on cost per token, which pushes the crossover down - an always-on pod running near saturation is very hard for serverless to beat. But it also raises the bar for keeping that pod fed: you need enough steady traffic to sustain concurrency 32, or the throughput (and the cheap token price) evaporates. The deeper treatment of batching economics lives in LLM inference cost at scale.

There is a latency tax, too. In the same benchmark median time-to-first-token started at 82ms at concurrency 1 and climbed as batches grew. Cheaper tokens, slower first token - pick the point on the curve that fits your SLO.

The serverless counter-argument: cold starts

The per-token premium is only half the story. The historical knock on serverless is the cold start - the delay when a scaled-to-zero worker has to boot, pull an image, and load weights before it can serve. If that penalty is large, you either eat latency or keep workers warm (which erases the scale-to-zero savings and pushes you back toward always-on economics).

This is where the platform matters. Cerebrium's own engineering has gone after every layer of that penalty. In our benchmarks, Cerebrium memory snapshots reduced cold starts by an average of 71% versus running the same workloads without snapshots, with reductions as high as 88% on vLLM. We also reworked AWS node startup to cut machine boot time by 83%, and rebuilt container image distribution so a lazy-loading path downloads a ~15MB metadata bootstrap instead of unpacking a full 4.87GB image before the container can start. The net effect, as described on our platform, is serverless CPU/GPU cold starts in the 2-4 second range.

When cold starts shrink to a couple of seconds, the whole calculus shifts: you can run genuinely spiky traffic on serverless without a warm-pool tax, which keeps billed utilization - and therefore cost - low. The cold-start playbook for latency-sensitive workloads is detailed in serverless GPU cold starts for voice AI, and a head-to-head on the cost-versus-cold-start trade-off is in Modal vs RunPod LLM inference cost and cold starts.

A decision checklist for CTOs

1. Measure your real billed utilization. Not peak - the hourly average per GPU-equivalent of traffic. Above ~48% on comparable hardware, lean always-on; below it, lean serverless.

2. Compute cost per 1M tokens at your actual concurrency, not at concurrency 1. The 24x spread from the curve above is the single biggest lever.

3. Price the latency SLO. Higher concurrency is cheaper but slower to first token; cold starts add a one-time penalty on serverless. Confirm your platform's cold start fits your SLO before counting the scale-to-zero savings.

4. Model the traffic shape, not the average. A flat 50% load and a spiky 50% average behave completely differently - bursty traffic favors serverless even above the naive break-even.

5. Consider a hybrid. Reserved/always-on capacity for your predictable baseline, serverless for the spikes above it, usually beats either pure strategy.

The break-even is not a universal constant - it is a line you can calculate once you have your hourly rate, your throughput at your concurrency, and your honest utilization. Get those three numbers and the choice makes itself.