GPU Inference for AI Agent Workloads

Most GPU sizing advice assumes one request equals one inference. Agents break that assumption. A single user turn becomes a trajectory: decide, call a tool, wait for it, read the result, decide again. Ten steps is ordinary and fifty is not unusual.

That one structural difference is what makes gpu inference for ai agent workloads a distinct sizing problem. Nothing about the model changes. What changes is that every fixed cost in the request path stops being paid once and starts being paid per step.

Why does an agent cost more GPU time than a chatbot?

Because the overheads multiply. Take the per-step numbers we have measured on our own infrastructure and put them next to each other.

Only the middle column is measured. The third is arithmetic on it, the per-step figure multiplied by ten: we did not benchmark a ten-step run, and your own trajectory length is the number that matters. It is there to show the shape of the compounding, not to report a result.

The first row is the model doing the thing an agent exists to do. In our testing a tool decision costs 330 to 510ms of time-to-first-token, on hardware that sustains 60 to 115 tokens per second on a single B200. That is the irreducible part.

The second row is not. A queue-based worker stack pays a broker hop on every task, and Redis introduces 100 to 500ms of broker overhead per task. On a one-shot inference that is noise. Across ten steps it is between one and five seconds of pure waiting, which can exceed the model time it is wrapping. Co-locating the services instead puts internal routing at about 5ms, under a router whose p99 lookup is roughly 500 microseconds.

The lesson is not that queues are bad. It is that an architecture decision you can safely ignore for batch inference becomes the dominant term once you multiply it by trajectory length.

Why agent traffic breaks hourly GPU billing

Agent GPUs are idle a lot. While a tool call is in flight - a database read, a web fetch, a payment API - the GPU that will handle the next step is doing nothing. Utilisation is not low because you sized badly; it is low because waiting is what the workload does.

Hourly billing charges you for that waiting. Per-second billing with scale-to-zero does not, but only if capacity comes back fast enough to be worth releasing. Ours does: workloads spin up only when requests arrive and shut down immediately after, with new workers provisioned typically within 2 to 4 seconds. Where model load dominates that window, memory snapshots restore a running CUDA process rather than rebuilding it.

Agent traffic is also bursty in a way chat traffic is not, because one user action fans out into many model calls. That fan-out is real at production scale: during a launch, Tavus scaled to 150 containers, and Distil Labs runs up to 150 requests per second per model with several models live at once. Size for the burst and the tail, not the average.

Does a smaller model work better for agents?

Often, yes, and this is the largest single lever available. Most steps in a trajectory are not hard reasoning. They are routing, extraction and formatting - narrow tasks where a small fine-tuned model is both faster and cheaper, with the frontier model reserved for the steps that genuinely need it.

Our benchmarks with a customer doing exactly this are unambiguous: we reduced inference costs by 50% and increased accuracy from 83% to 92% at production scale. Cheaper and more accurate at once is unusual, and it happens because a task-specific model is not paying for generality it never uses. Because that saving lands on every step, a per-step improvement compounds across the trajectory in a way a one-shot workload never sees. If you are choosing where to start, our LLM deployment notes cover the serving side, and a worked LangChain tool-calling agent shows the loop end to end.

One practical constraint: agent steps carry accumulated context forward, so payload limits bind sooner than people expect. Celery tasks degrade beyond about 100 MB, against 1 GB natively here.

How should I size a GPU fleet for agents?

Start from the trajectory, not the request. Measure your median and p95 step count, multiply by your measured per-step latency, and you have a real turn-latency budget. Then decide what fraction of steps can move to a smaller model. Then size for burst concurrency rather than mean throughput, and check that your scale-up time is short enough that scaling to zero between bursts is actually worth doing. Current rates are on the pricing page; the figures here come from our own published benchmarks and will drift, so re-derive before you budget against them.

The short version

Agent workloads pay every fixed overhead once per step. Cut the per-step overheads first, because they multiply: co-locate the services to turn a 100 to 500ms broker hop into about 5ms, move the narrow steps onto a small fine-tuned model, and bill by the second so the idle time during tool calls is free. Then size for the burst.