NVIDIA B200 Inference: Measured Throughput and TTFT
Connor Blier
Founding GTM
NVIDIA B200 Inference: Measured Throughput and TTFT
On a single NVIDIA B200 in a production pipeline we measure 60 to 115 tokens per second with a time-to-first-token of 300 to 510 milliseconds. Those two numbers move almost independently of each other, and which one you optimise for should decide how you configure the machine.
Published numbers for nvidia b200 inference throughput ttft are still scarce, largely because the chip is new enough that most figures in circulation are vendor slides or synthetic loops. Ours come out of a running voice-agent pipeline, which makes them less flattering and more useful.
In our testing, a single B200 sustains 60–115 tokens per second at 300–510ms time-to-first-token. On the same hardware, a tool-decision call lands at 330–510ms TTFT. That second figure is the one that matters for agentic workloads, because a tool decision sits directly in the user's perceived response path.
Throughput and TTFT are different problems
It is tempting to treat them as one axis. They are not. Time-to-first-token is dominated by prefill, scheduling and network placement; sustained tokens per second is dominated by memory bandwidth and batching. You can improve one and regress the other, and most configuration changes do exactly that.
Batching is the clearest example. In our serving-framework benchmark SGLang reached 460 tokens per second at batch size 64 — far above single-stream throughput — while the best time-to-first-token in the same benchmark was 123ms from vLLM at batch size 1. Larger batches raise throughput and raise latency. There is no configuration that wins both.
That benchmark used 256 input tokens on a single H100 at batch size 1, measuring total roundtrip to the platform. We state the methodology because a tokens-per-second number without its input length, batch size and precision is not a measurement — it is a marketing figure, and it is why published benchmarks disagree so wildly.
Why our B200 number looks lower than numbers you have seen
Here is the comparison that makes the point. A single Ampere A10 — a far smaller, far older GPU — reaches about 1700 output tokens per second at FP8 running Llama 3 8B. The same A10 at FP16 does about 600 tokens per second at $0.000555 per second.
So a small GPU posts a number an order of magnitude above our B200 figure. Nothing is wrong with either measurement. They are different models, different precisions and different batch settings. The 8B model is small enough to be bandwidth-cheap per token; the model behind the B200 numbers is not. Tokens per second is a property of the model-hardware-precision triple, never of the GPU alone, and any comparison that omits the other two is meaningless.
| Configuration | Throughput | TTFT | What it tells you |
|---|---|---|---|
| Single B200, production pipeline | 60–115 tok/s | 300–510ms | Realistic large-model serving |
| A10, Llama 3 8B, FP8 | ~1700 tok/s | — | Small models are bandwidth-cheap |
| A10, FP16 baseline | ~600 tok/s | — | Precision alone moves throughput ~3x |
| H100, SGLang, batch 64 | 460 tok/s | — | Batching buys throughput |
| H100, vLLM, batch 1 | — | 123ms | Framework choice moves TTFT |
Turning the numbers into cost
Throughput only becomes decision-relevant when you divide it into a rate. The B200 is published at $0.001679 per GPU-second. At 60–115 tokens per second that is roughly $0.000015 to $0.000028 per output token on a single stream — before batching, which is the main lever for improving it.
The comparison to make is not B200 against A10 on tokens per second. It is cost per token at your required latency. If your product tolerates 500ms time-to-first-token, batching hard on fewer, larger machines is usually cheaper. If you need to stay under 300ms, you are buying latency headroom and should expect to pay for idle capacity to get it. We worked the full version of that arithmetic in deploying AI workloads on serverless GPUs for global scale.
What else moves these numbers
Placement. For interactive workloads, region choice moves user-perceived latency more than chip choice does. Our global-scale voice architecture has the per-stage breakdown.
Cold start. A B200 you are waiting on delivers zero tokens per second. If traffic is spiky, memory snapshots and fast restore affect your real average throughput more than peak benchmark numbers do.
Precision. FP8 versus FP16 moved the A10 by roughly 3x in our own measurements. Quantisation is usually the cheapest throughput win available.
How to benchmark this yourself
Fix and publish your methodology first — input length, batch size, precision, and whether you are timing the model or the full roundtrip. Change one variable at a time.
Measure TTFT and sustained throughput separately. A single "tokens/sec" number hides which one regressed.
Measure at your batch size, not the vendor's. Benchmarks are usually run at whatever batch maximises the published figure.
Divide by the rate. Cost per token at your latency target is the only number that makes two GPUs comparable.
Current per-second rates for the GPU classes here are on the pricing page; the 2026 GPU buyer's guide has the cross-provider table, and the serverless GPU provider comparison is the shortlist version.
The short version
A single B200 gives us 60–115 tokens per second at 300–510ms TTFT on a real production model. Do not compare that to a small-model figure from a smaller GPU — compare cost per token at the latency your product actually needs.
Frequently asked questions
- What throughput and TTFT does an NVIDIA B200 achieve for inference?
- In our production pipeline a single B200 sustains 60 to 115 tokens per second with a time-to-first-token of 300 to 510 milliseconds. On a tool-decision call the same hardware measures 330 to 510ms TTFT. Those figures come from a running workload rather than a synthetic benchmark loop.
- Why do some smaller GPUs report higher tokens per second than a B200?
- Because tokens per second is a property of the model, hardware and precision together, not of the GPU. An Ampere A10 reaches about 1700 output tokens per second at FP8 on Llama 3 8B, well above our B200 figure on a much larger model. Any throughput number without its model size, precision and batch size is not comparable.
- Does batching improve throughput and latency together?
- No, they trade against each other. In our benchmark SGLang reached 460 tokens per second at batch size 64, while the lowest time-to-first-token, 123ms from vLLM, came at batch size 1. Larger batches raise throughput and raise latency, so you configure for whichever your product is bound by.
- What does B200 inference cost per token?
- The B200 is published at $0.001679 per GPU-second. At 60 to 115 tokens per second that works out to roughly $0.000015 to $0.000028 per output token on a single stream, before batching. Batching is the main lever for improving it, at the cost of latency.
- How should I benchmark GPU inference fairly?
- Fix and publish the methodology first: input length, batch size, precision, and whether you are timing the model or the full roundtrip. Measure time-to-first-token and sustained throughput separately, run at your own batch size rather than the vendor's, and compare cost per token at your latency target.
Sources
- cerebrium.ai
“In production we measure **60–115 tokens/sec** and **300–510 ms TTFT** on a [single NVIDIA B200](https://cerebrium.ai/docs/getting-started/introduction).”
The headline measurement: sustained generation rate and time-to-first-token for a single B200 in a production pipeline, not a synthetic loop.
- cerebrium.ai
“| LLM #1 — tool decision (TTFT) | 330–510 ms |”
The same hardware measured on a tool-decision call, which is the latency users actually feel in an agentic pipeline.
- cerebrium.ai
“| NVIDIA B200 | 180–192 GB | $0.001679 | $0.001636 | $0.001736 | $0.002772 |”
Published per-GPU-second rate for the B200, needed to turn a throughput number into a cost-per-token figure.
- cerebrium.ai
“In this tutorial we will achieve ~1700 output tokens per second (FP8)on a single Nvidia A10 instance however you can go up to ~4500 output tokens per second on a single Nvidia A100 40GB instance or even ~19,000 tokens on a H100.”
A much smaller GPU posting a far larger tokens/sec number on a small model — the single clearest demonstration that throughput figures are not comparable across model sizes.
- cerebrium.ai
“an Ampere A10 had a throughput of ~600 tokens per second (FP16) at a cost of $0.000555 per second.”
The same A10 at FP16 rather than FP8, isolating precision as a variable in throughput comparisons.
- cerebrium.ai
“As for throughput use cases, it seems SGLang was a clear winner achieving a throughput of 460 tokens per second on a batch size of 64.”
How much batching changes the throughput number on identical hardware.
- cerebrium.ai
“From our tests, it seems that vLLM is the best framework for use cases looking for the lowest TTFT of 123ms which is extremely quick if you take into consideration network latency.”
Best-case time-to-first-token from our serving-framework benchmark, showing framework choice moves TTFT independently of the GPU.
- cerebrium.ai
“The following benchmark was done with 256 input tokens on 1xH100 of batch size 1. This was the total roundtrip time of a request made to the Cerebrium platform.”
The methodology behind those framework numbers — stated because a throughput figure without its input length and batch size is not a measurement.
