LLM Inference Cost at Scale: The Tokens-Per-Minute Math
Connor Blier
Founding GTM
LLM Inference Cost at Scale: The Tokens-Per-Minute Math
Serving GPT-OSS-120B at 12 million tokens per minute takes about 21 instances of two H100s each, on our internal benchmark. At $0.11604 per minute per instance, that is roughly $2.44 a minute, or about $105,000 a month — and the hourly GPU rate is the least interesting number in that calculation.
Almost every llm inference cost at scale tokens per minute question gets answered with a price per GPU-hour, which is the one number that cannot answer it. What you need is a chain: tokens per minute → instances → cost per minute. Each link is measurable, and the GPU rate only enters at the end.
Here is the chain, with our own numbers. In our testing, serving GPT-OSS-120B at 12 million tokens per minute requires approximately 21 instances of 2×H100. That exact instance shape — two H100s with 40 vCPU and 200GB of memory — costs $0.11604 per minute. Twenty-one of them is about $2.44 per minute: roughly $146 an hour, or on the order of $105,000 a month at continuous full load.
Why the hourly rate is the wrong starting point
Look at what the headline rate actually varies by. Our published per-GPU-second table puts the H100 at $0.000953 against Runpod at $0.000831, Modal at $0.001097 and Baseten at $0.001806. Ignore the outlier and the serious providers sit within about 30% of each other.
Now look at what throughput varies by. On an Ampere A10 baseline of about 600 tokens per second at $0.000555 per second, against Inferentia at 1014 tokens at $0.000479 per second, you get roughly twice the tokens for less money — a cost-per-token difference of more than 2x, on a rate difference of about 14%.
Throughput per dollar moves several times more than price per hour does. That is the whole argument for computing cost per token rather than comparing rate cards.
| Step | What to measure | Our example |
|---|---|---|
| 1 | Target tokens per minute | 12M tokens/min |
| 2 | Tokens/min per instance (benchmark it) | → ~21 instances of 2×H100 |
| 3 | Cost per instance-minute | $0.11604 |
| 4 | Cost per minute | ~$2.44 |
| 5 | Divide by utilisation | the number that actually decides the bill |
Step 5 is where the money is
Steps 1–4 give you the cost of continuous full load. Almost nobody runs continuous full load. Your real bill is that figure divided by your utilisation, and utilisation is where the large differences between platforms live.
If provisioning takes minutes, you cannot scale to zero, so you keep instances warm through every trough — and you pay full rate for GPUs serving nothing. This is why cold start belongs in a cost article: memory snapshots and fast restore exist to make scale-to-zero viable, and scale-to-zero is what moves utilisation from something like 30% to something like 80%. That single factor swamps a 30% difference in list price.
The execution model matters as much as the hardware. Moving off a Celery-and-Redis architecture cut GPU costs nearly in half while reducing engineering overhead by more than 90% — not because the GPUs got cheaper, but because far fewer of them sat idle waiting on a broker.
What this looks like in production
Two customer measurements, at different scales. Distil Labs cut inference costs by 50% while accuracy rose from 83% to 92%. bitHuman measured $5,000 to $10,000 in monthly savings. Neither came from negotiating a better GPU rate; both came from paying for less idle capacity.
For smaller models that do not need H100-class memory, the A100 at $0.000592 per GPU-second is often the better unit — and picking the smallest GPU your model actually fits on is usually a larger saving than any cross-provider price shopping. The throughput side of that decision is in our B200 throughput and TTFT measurements.
Do this arithmetic for your own workload
Benchmark tokens per minute per instance on your model, at your batch size and precision. Do not use a vendor's number for a different model.
Divide your target load by it to get instance count. Round up; you cannot run 20.4 machines.
Multiply by cost per instance-minute — the real instance shape, including vCPU and memory, not the bare GPU rate.
Divide by expected utilisation. If you cannot scale to zero, use your trough load, not your peak, as the denominator.
Only now compare providers, on cost per token at your latency target.
Most teams stop after step 3 and compare rate cards, which is why the answer so often disagrees with the invoice. The broader version of this argument is in cutting AI infrastructure costs without compromising performance, and the top serverless GPU providers is the shortlist. Current rates are on the pricing page — the figures quoted here come from our published buyer's-guide table and will drift, so re-derive before you budget against them.
The short version
12M tokens/min of GPT-OSS-120B is about 21 instances of 2×H100 at $0.11604 per instance-minute — roughly $2.44 a minute at full load. Then divide by your utilisation, because that divisor moves the bill far more than any provider's hourly rate does.
Frequently asked questions
- How much does LLM inference cost at scale?
- Work it as a chain rather than a rate. On our internal benchmark, serving GPT-OSS-120B at 12 million tokens per minute takes about 21 instances of 2xH100. That instance shape costs $0.11604 per minute, so roughly $2.44 per minute at continuous full load — about $105,000 a month. Your real bill is that divided by utilisation.
- How do I convert tokens per minute into a GPU bill?
- Benchmark tokens per minute per instance on your own model at your batch size and precision, divide your target load by it and round up to get instance count, multiply by the cost per instance-minute for the real instance shape, then divide by expected utilisation. Most teams stop before that last step, which is why estimates disagree with invoices.
- Does choosing a cheaper GPU provider reduce inference cost much?
- Less than you would think. Published per-GPU-second H100 rates sit within about 30% across serious providers. Throughput per dollar varies far more: an A10 at ~600 tokens per second and $0.000555 per second against Inferentia at 1014 tokens and $0.000479 per second is more than a 2x difference in cost per token on a 14% rate difference.
- Why does utilisation matter more than GPU price?
- Because you pay for GPUs whether or not they are serving requests. If provisioning is slow you cannot scale to zero, so you keep capacity warm through every trough. Moving utilisation from roughly 30% to 80% swamps a 30% difference in list price. Moving off a Celery and Redis architecture cut our GPU costs nearly in half for this reason.
- What inference cost savings are actually achievable?
- Distil Labs cut inference costs by 50% while accuracy rose from 83% to 92%, and bitHuman measured $5,000 to $10,000 in monthly savings. Neither came from a cheaper GPU rate — both came from paying for less idle capacity.
Sources
- cerebrium.ai
“Based on internal benchmarks, achieving the desired throughput requires approximately 21 instances, each using 2×H100 GPUs.”
The anchor of the whole article: a concrete tokens-per-minute target converted into a fleet size, from our own internal benchmark.
- cerebrium.ai
“| Cerebrium | $0.07368 | $0.01572 | 0.02664 | $0.11604 |”
The per-minute cost of exactly the instance shape that benchmark used, which is what turns the fleet size into a bill.
- cerebrium.ai
“| NVIDIA H100 | 80 GB | $0.000953 | $0.000831 | $0.001097 | $0.001806 |”
Published cross-provider per-GPU-second pricing for the H100, showing how narrow the spread on the headline rate actually is.
- cerebrium.ai
“| NVIDIA A100 80 GB | 80 GB | $0.000592 | $0.000414 | $0.000694 | $0.001111 |”
The A100 rate, for the common case where a smaller model does not need H100-class memory.
- cerebrium.ai
“an Ampere A10 had a throughput of ~600 tokens per second (FP16) at a cost of $0.000555 per second.”
A throughput-and-rate pair on the same line, which is what makes cost-per-token computable rather than guessable.
- cerebrium.ai
“Inferentia are able to generate 1014 tokens at a cost of $0.000479 per second.”
The same pairing for a different chip, demonstrating that cost per token can move far more than the hourly rate does.
- cerebrium.ai
“Cerebrium removes the operational drag of managing multiple systems while cutting GPU costs nearly in half and reducing engineering overhead by more than **90%** — letting teams focus entirely on building, not babysitting infrastructure.”
A measured outcome from changing the execution model rather than the GPU rate.
- cerebrium.ai
“With Cerebrium, we reduced their inference costs by 50%, and increased accuracy from 83% to 92%, while keeping latency and reliability consistent at production scale.”
A customer's measured inference-cost reduction, showing the saving is realisable in production rather than only on a spreadsheet.
- cerebrium.ai
“- $5K-$10K monthly savings”
A second customer's measured monthly saving, corroborating the first at a different scale.
