Where Latency Goes in an AI Inference Pipeline
Connor Blier
Founding GTM
Where Latency Goes in an AI Inference Pipeline
Most teams tuning an inference pipeline optimise the model first. In our own measurements the largest single term is usually geography: a London user calling us-east-1 pays 150 to 250 milliseconds before any model runs. Moving the compute closer cut that to 30 to 70 milliseconds.
When a team asks where to find network latency ai inference pipeline savings, the instinct is to reach for the model — quantise it, batch it, swap the serving framework. That work is real, but it is usually second-order. The budget is dominated by where things are and how many hops sit between them.
In our measurements the geography term is the big one. A user in London calling a service in us-east-1 pays 150 to 250 milliseconds of pure network latency before any computation begins. When we placed compute near users instead, the same measurement fell to 30 to 70 milliseconds, a 60% decrease. No model changed.
The five places latency actually hides
Work through them in order of size, because that is the order in which fixing them pays.
1. Geography. Covered above, and it is almost always the largest avoidable term. It is also the one teams most often accept as fixed, because the region was chosen once, early, by someone else.
2. Model API hops. Calling a hosted model over the public internet, rather than running it next to your orchestration, costs more than most teams expect. We measured an external LLM API at ~800ms against ~100ms for a locally hosted model on the same workload. For voice specifically, the per-stage version of this comparison is in our global-scale voice architecture, where co-locating speech models bought back roughly 140ms on transcription and 90ms on synthesis.
3. Queue and broker overhead. If inference sits behind a conventional task queue, the broker is in the request path. Redis adds 100 to 500 milliseconds of broker overhead per task — a range wide enough to dominate a real-time budget on its own, and invisible in any model-level profile.
4. Internal routing. This one is usually not your problem, and it is worth knowing so you stop looking. Internal routing on our platform costs about 5 milliseconds, and inter-cluster routing measures about 2 milliseconds while saving over 150ms by keeping traffic off longer paths. Our request router runs at a p99 lookup latency of roughly 500 microseconds and routed requests land within milliseconds of direct cluster calls. A routing layer built properly does not show up in the budget.
5. Cold start. Not latency in the networking sense, but it is latency to the user, and it is the term that ruins tail percentiles. If a request arrives when nothing is warm, the user waits for provisioning. Memory snapshots and fast restore is how we attack it; the effect shows up in p99, never in p50.
| Source | Typical cost | Fix |
|---|---|---|
| Wrong region | 150–250ms | Deploy near users (→ 30–70ms) |
| Hosted model API hop | ~800ms vs ~100ms local | Co-locate the model |
| Task-queue broker | 100–500ms per task | Take the broker out of the request path |
| Internal routing | ~2–5ms | Usually already fine — do not start here |
| Cold start | seconds, in the tail | Snapshots, or keep capacity warm |
Why profiling the model finds none of this
A model-level profiler measures the model. Every term in the table above except the last happens outside it, which is why a pipeline can look healthy in profiling and feel slow to users. The symptom is a large, stable gap between your measured inference time and your measured end-to-end time — and that gap is the whole subject of this article.
Measure end-to-end from the client, not from the service. If the two numbers disagree by hundreds of milliseconds, the difference is in the table, not in the model.
The tail matters more than the mean for anything interactive
Averages hide the failure. A pipeline averaging 200ms with a p99 of 3 seconds feels broken to the one user in a hundred who hits it, and in a conversational product that user notices immediately. This is why cold start belongs in a latency discussion at all, and why availability belongs next to it: Creatium runs at 99.999% uptime on this stack, which is the precondition for any latency claim being worth anything.
For a worked example of the whole budget assembled, our Ultravox pipeline reaches 600ms end-to-end first-time-to-audio, and a low-latency architecture with live web retrieval shows what happens when you add retrieval stages. If your constraint is concurrency rather than latency, concurrent voice sessions per GPU covers how the two interact.
How to find your own biggest term
Measure end-to-end from the client, in the region your users are actually in. Not from a developer laptop next to the datacentre.
Subtract model time. Whatever remains is network, queueing and routing — the terms above.
Attack in size order. Region first, then API hops, then the broker. Internal routing last, if ever.
Report p99, not the mean. For interactive workloads the tail is the product.
Per-second rates for the GPU classes referenced here are on the pricing page.
The short version
Geography and hops beat model tuning for most inference pipelines. A misplaced region costs 150–250ms, a hosted API hop can cost 700ms, and a task broker can cost 500ms — while internal routing costs about 5ms and is rarely worth your attention.
Frequently asked questions
- Where does latency come from in an AI inference pipeline?
- Mostly outside the model. In our measurements the largest avoidable term is geography — a London user calling us-east-1 pays 150 to 250ms before any computation — followed by hosted model API hops, task-queue broker overhead of 100 to 500ms, and cold start in the tail. Internal routing costs about 5ms and is rarely the problem.
- How much latency does deploying closer to users actually save?
- In our multi-region measurements, moving compute near users took a London to us-east-1 path from 150-250ms down to 30-70ms, a 60% decrease, with no model changes at all. Region placement moves user-perceived latency more than chip or framework choice for anything interactive.
- Is it faster to self-host a model than to call a hosted API?
- On the same workload we measured an external LLM API at roughly 800ms against about 100ms for a locally hosted model. For voice pipelines, co-locating speech models saved roughly 140ms on transcription and 90ms on synthesis. The saving comes from removing a public-internet hop, not from a better model.
- Does a task queue add latency to inference?
- Yes, and it is often overlooked because it sits outside the model. Redis adds 100 to 500 milliseconds of broker overhead per task, which is enough to dominate a real-time budget on its own and does not appear in any model-level profile.
- Why does my pipeline profile fast but feel slow?
- Because a model profiler measures the model, and most latency happens outside it. The symptom is a large stable gap between measured inference time and measured end-to-end time. Measure end-to-end from the client in your users' region, subtract model time, and whatever remains is network, queueing and routing.
Sources
- cerebrium.ai
“Previously, if you were in London and making requests to our servers in us-east-1, you’d experience 150–250ms of network latency.”
The cost of getting the region wrong: pure geography, before any model has run. The single largest avoidable term in most interactive pipelines.
- cerebrium.ai
“Now, by deploying directly in the UK, that latency drops to just 30–70ms - a 60% decrease!”
What placing compute near the user does to that same number — the measured result of fixing the geography rather than the model.
- cerebrium.ai
“**Ultra-low internal latency:** Internal routing operates at **~5 ms latency**, enabling near-instant request dispatch.”
Routing overhead inside the platform, establishing the floor that internal hops cost.
- cerebrium.ai
“By deploying them inside Cerebrium, we take advantage of inter-cluster routing between the services that allows them to incur a “network request latency” of ~2ms, which alone saves us 150ms+.”
Inter-cluster routing measured on our own fleet, and the saving it produced — the clearest single demonstration that placement dominates.
- cerebrium.ai
“Redis introduces 100-500ms of broker overhead per task, which adds up in low-latency pipelines.”
What a conventional task-queue broker contributes, for teams whose inference sits behind Celery and Redis rather than a direct call path.
- cerebrium.ai
“This does come with a latency trade-off (~800ms vs ~100ms locally).”
The difference between calling a hosted model API and co-locating the model, measured on the same workload.
- cerebrium.ai
“Lookups have a p99 of roughly 500 microseconds, which allows us to spend more of that budget making routing decisions.”
Tail-latency figure for our request router, showing that a well-built routing layer is not a meaningful contributor to the budget.
- cerebrium.ai
“With DNS, TLS, and the connection kept warm, routed requests were within milliseconds of direct cluster calls across the customer’s core endpoints.”
Corroborates the same point from the other direction: routed and direct calls are indistinguishable in practice.
- cerebrium.ai
“This freed the development team to focus almost entirely on improving the core product, while the platform continues to deliver 99.999% uptime and cost efficiency—providing the reliability and financial predictability needed to scale.”
Availability context for the routing layer, since a latency argument is worthless if the path it recommends is not reliable.
