Batch vs Real-Time LLM Inference Cost Per Million Tokens

The one-line answer

Batch is the cheaper tier for cost per million tokens whenever your work can wait, because batch requests run on fully-busy workers and are billed at a discount, while real-time pays a premium to keep latency-sensitive capacity on standby. On Runpod, batch requests are billed at up to 30% off on-demand pricing (20% off on H100, H200, B200 and B300). Real-time serverless, by contrast, can cost about 2.3x as much per token as a persistent pod on the same GPU when the worker stays busy. So the question batch vs real time llm inference cost per million tokens is really a question about one number: how busy your workers are.

Cost per million tokens is a utilization number, not a sticker price

The trap is reading provider price pages as if cost per million tokens were fixed. It is not. Cost per token is the hourly GPU rate divided by tokens produced in that hour, and tokens produced depends entirely on batch size and how continuously the worker runs. We worked the full chain through in our LLM inference cost at scale guide, and the per-request version in token usage and cost-per-request tracking.

The mechanism matters because throughput scales hard with batch size. In our benchmarks of Llama 3.1 70B on a single H100, SGLang reached 460 tokens per second at a batch size of 64 - far more total tokens per GPU-hour than the same hardware answering one request at a time. A batch job keeps the GPU at that high-batch operating point continuously; a latency-bound real-time endpoint rarely does. Same GPU, same hourly rate, wildly different cost per million tokens.

The real-time premium: why busy serverless costs ~2.3x more per token

Real-time inference buys you responsiveness, and you pay for it twice: once for idle standby, and once in a higher per-token rate even when busy. Runpod measured that a busy Serverless worker cost about 2.3 times as much per token as a Pod on the same GPU in an A40 benchmark. That premium is the price of elasticity - the ability to scale to zero and absorb spikes.

That does not make serverless wrong. It makes it situational. Scale-to-zero is exactly what you want for spiky, interactive traffic, and the cold-start cost can be small: in our Llama 3 8B TensorRT-LLM tutorial the model loaded into GPU memory in roughly 10-15 seconds per cold start. If you want to serve real-time traffic the right way, see autoscaling Llama 70B inference and small language model inference cost per request.

The batch discount: batch is a distinct pricing tier

Batch is not just "real-time, slower." It is a separate economic product. Batch and long-context are now tracked as distinct pricing modes in LLM API pricing datasets, alongside caching - which is the market's way of saying the per-token rate genuinely differs.

The discount is explicit on self-hosted-style platforms too: Runpod's batch requests are billed at 20% off on H100, H200, B200 and B300, and up to 30% off depending on GPU type. Stack that stated discount on top of the higher batch-size throughput above and you get the familiar 2-3x gap for offline workloads: embeddings, document enrichment, evals, and synthetic data generation. For transcription, we break the same math down per audio-hour in Whisper batch transcription cost.

The breakeven line: where real-time stops being worth it

Here is the pivot the sticker prices hide. Runpod found Serverless costs less than Pods only when its flex workers would be billed for under about 29 minutes of each hour. That is the line. Below roughly 48% utilization, elastic real-time wins because you stop paying for idle. Above it, a persistent pod - or a batch tier - is cheaper, because you are paying the ~2.3x busy-worker premium on a worker that is almost always busy anyway.

So the decision inverts around utilization:

• Worker busy < ~29 min/hour: real-time serverless with scale-to-zero is cheapest. Pay per request, not per idle minute.

• Worker busy > ~29 min/hour: you are effectively always-on. A persistent pod avoids the serverless premium, and a batch tier adds the explicit 20-30% discount on top.

(The 29-minute figure is from Runpod's A40 test; the exact minute shifts with GPU and model, but the shape holds.) For the always-on case, see reserved GPU capacity for production LLM inference.

A decision framework for the Lead ML Engineer

1. Classify the latency requirement first. If a human or a live system is waiting, it is real-time. If results can land minutes or hours later, it is batch - and batch is a different pricing game.

2. Estimate worker utilization, not request volume. Volume alone is misleading; plot how many minutes per hour a worker is actually billed. Compare against the ~29-minute line.

3. For real-time, drive batch size up within your latency budget. Higher effective batch at serving time is the single biggest lever on cost per million tokens - see vLLM server parameters for throughput.

4. For batch, claim the explicit discount tier. Do not run offline jobs on real-time endpoints; the 20-30% is left on the table otherwise.

5. Watch cold starts if you scale to zero. A 10-15s model load is cheap for batch and acceptable for many real-time apps; measure it against your SLO.

The payoff is real. We measured a 50% drop in inference cost for one customer: DistilLabs reduced inference costs by 50% and raised accuracy from 83% to 92% while holding latency and reliability at production scale - largely by matching workload shape to the right serving tier. If you are still choosing a platform, our Modal vs Runpod cost and cold-start comparison carries the vendor-level detail.