vLLM on TPU vs H100: Cost & Throughput Guide

The question of vLLM on Google TPU vs H100 GPU for LLM inference cost and throughput used to settle itself on a technicality: vLLM was a GPU-first project, so the TPU path meant a different serving stack, a different feature set, and a migration no VP of Engineering wanted to own. That technicality is gone. With TorchTPU, vLLM now runs natively on TPU with the same flags you already use on GPU, which turns this into a straight buyer decision about price-performance on your workload rather than a compatibility gamble.

This guide frames that decision the way we'd run it internally: what actually changed on the TPU side, the orchestration layer that now spans both accelerators, the first-hand H100 vLLM numbers we've measured as a GPU baseline, and a framework you can run on your own traffic. One guardrail up front, because it matters for how you read this: there is no honest head-to-head TPU-vs-H100 cost table to publish yet. The concrete throughput, cold-start, and cost figures below are all GPU-side measurements. The TPU side is a capability story. Treat anyone selling you a clean "Nx cheaper" number with suspicion until they show you the workload it was measured on.

The real question isn't TPU vs H100 - it's vLLM-on-TPU vs vLLM-on-GPU

Framing the choice as "TPU vs H100" invites a spec-sheet war that almost never predicts production cost. What you actually buy is a serving stack on an accelerator. Now that the serving stack can be identical on both sides, the variables that decide cost and throughput are the ones you control: model size, quantization, batch size, context length, concurrency, and how well your parallelism strategy maps to the hardware. The accelerator is one input, not the verdict.

That reframing is good news for a VP Engineering. It means you can keep one inference codebase, benchmark it on both targets, and let your own numbers - not a vendor's best case - pick the winner per workload.

What changed: vLLM now runs natively on TPU (TorchTPU)

Feature parity, not a subset

The historical objection to TPU inference was that you gave things up. With TorchTPU that's no longer true. The vLLM features production teams rely on carry over: in the project's words, "Speculative decoding, prefix caching, multi-LoRA, structured outputs, and multimodal inputs are all enabled with the usual vLLM flags." For a buyer, parity is the whole point - it means the serving behaviour you tuned on GPU isn't something you have to re-earn on TPU.

Six parallelism dimensions

TorchTPU also exposes vLLM's full parallelism surface. "All six of vLLM's parallelism dimensions are supported, on both single-host and multi-host TPU slices, and they can be combined" - Tensor, Data, Pipeline, Expert, Prefill Context, and Decode Context parallelism. That breadth matters because the right parallelism mix is one of the biggest levers on throughput-per-dollar, and having all six available means the hardware isn't constraining your sharding strategy.

Why this removes the old objection

Feature parity plus full parallelism means moving a vLLM workload to TPU is no longer a rewrite - it's a target change. The compatibility risk that used to make the comparison moot has collapsed into a cost-and-throughput question you can actually measure. The parallelism and server-side tuning that drive those numbers are the same discipline on either accelerator; our deep dive on vLLM server parameters and inference throughput covers the flags that matter most.

The orchestration layer: Ray / Anyscale native PyTorch on TPU

Serving is only half the stack; you still have to schedule it. Here too the gap closed. "Today Anyscale and Google Cloud are announcing native Ray support for PyTorch on Google TPUs, the foundational supercomputing infrastructure powering Gemini and large-scale AI workloads on Google Cloud." That lets distributed training and inference run across both GPUs and TPUs on a single control plane.

Crucially, the programming model converged: "Following TorchTPU, we've evolved Ray to run one worker per TPU chip, matching the single-device-per-worker model that PyTorch developers are used to on GPUs." For teams already on Ray Serve, this means the orchestration mental model doesn't fork by accelerator. If you're weighing that layer specifically, see our notes on Anyscale and Ray Serve alternatives.

The GPU baseline: first-hand H100 vLLM cost and throughput

You can't evaluate TPU economics without a GPU reference point you trust. These are Cerebrium's own numbers, which is why we lead with them.

Throughput

On Llama 3.1 70B FP8 on a single H100, in our testing SGLang was the throughput winner, "achieving a throughput of 460 tokens per second on a batch size of 64." On smaller models the per-card ceiling is higher: for Llama 3 8B we measured "~1700 output tokens per second (FP8) on a single Nvidia A10" (the A100 and H100 figures cited alongside it are NVIDIA's published TensorRT-LLM extrapolations, not ours). The headline: throughput is a function of model, engine, quantization, and batch size - not the card alone.

Cold start

Cold start is a real cost line for bursty or scale-to-zero traffic. We measured a TensorRT-LLM Llama 3 8B engine where loading "the model into GPU memory" takes "roughly ~10-15s" on every cold start. Larger weights push that higher, which is why replica startup is its own optimization problem - our writeup on autoscaling Llama 70B replica startup goes into the mitigations.

Cost

Cost is where workload shape dominates. One customer running single-digit-million requests/day: "With Cerebrium, we reduced their inference costs by 50%, and increased accuracy from 83% to 92%, while keeping latency and reliability consistent at production scale." That gain came from serving architecture and autoscaling, not a different chip - a useful reminder that accelerator choice is one of several cost levers. For the full picture, see LLM inference cost at scale.

A cost-and-throughput evaluation framework (run this on your own workload)

Metrics that matter

Measure four things per candidate (TPU and H100): sustained output tokens/sec at your target batch size, time-to-first-token at your concurrency, cold-start time, and cost per million output tokens. Our H100 time-to-first-token numbers, for context, were measured "with 256 input tokens on 1xH100 of batch size 1" as "the total roundtrip time of a request made to the Cerebrium platform" - specify your own input/output shape the same way, because averages hide the tail.

Normalize before you compare

Never compare raw tokens/sec across accelerators. Normalize to cost per million output tokens at a fixed quality bar and a fixed latency SLO. A card that's faster but priced higher can lose on cost; a cheaper card that forces a smaller batch to hold your SLO can lose on throughput. Quantization shifts the whole curve - see 4-bit LLM inference on H100 for how much.

Where parallelism flexibility pays off

For large or MoE models, the parallelism mix is often worth more than the accelerator choice. Because TorchTPU exposes all six vLLM parallelism dimensions, a sharding strategy that's awkward on one target may be clean on another - test the actual combination, not the defaults. Multi-node LLM inference on serverless clusters covers the multi-host case.

Decision guide: when to lean TPU, when to lean H100

Lean toward evaluating TPU when you already run heavily on Google Cloud, your workload is large enough to justify multi-host slices, and your sharding strategy benefits from vLLM's parallelism breadth - the compatibility tax that used to rule TPU out is paid off. Lean toward H100 when you want the deepest ecosystem of tuned engines (vLLM, SGLang, TensorRT-LLM), when your traffic is bursty and cold start dominates, or when you simply have trusted H100 numbers in hand and no TPU baseline yet. In every case, the honest answer comes from running the framework above on your own traffic. For the broader hardware landscape, our 2026 GPU buyer's guide and H100 vs H200 throughput guide are the companion reads.