AWS Alternatives for AI Workloads: What Actually Changes
Connor Blier
Founding GTM
AWS Alternatives for AI Workloads: What Actually Changes
Most teams looking at AWS alternatives for AI workloads are not actually shopping for a cheaper GPU hour. They are trying to escape a provisioning model that takes minutes to hand them a machine. In our own testing, an EC2 instance took two to seven minutes to become ready to run containers.
The question is usually framed as a price comparison, and that framing is what makes it hard to answer well. Compare GPU hourly rates across providers and the numbers land close enough that the exercise feels pointless. The reason teams actually migrate has very little to do with the hourly rate, and quite a lot to do with how long it takes to get a machine and what you have to pay for while you wait.
In our testing, the gap that matters is provisioning time, not price. We ran a general-purpose cloud fleet ourselves before rebuilding it, and the numbers are unflattering: from the moment we requested a machine to the moment it was ready to run user containers, the process could take anywhere from two to seven minutes. The mechanical cause was mundane — booting a machine required loading more than 17 GB of platform container images. That is not an AWS defect. It is what a general-purpose instance primitive costs when you point it at AI-sized images.
Why the provisioning unit is the real problem
If a machine takes minutes to arrive, you cannot scale to zero. If you cannot scale to zero, you keep GPUs warm through every trough in your traffic, and idle GPU capacity is the single largest avoidable line item in most inference budgets. The hourly rate is a multiplier on a number that provisioning latency has already decided for you.
This is why the honest comparison is not "what does an H100 cost here versus there" but "what fraction of the H100s I pay for are serving requests". A platform with a two-second cold start and a slightly higher list rate can be dramatically cheaper in practice than a slower one with a keener sticker price, because it lets you stop paying for the troughs.
We spent a lot of engineering on exactly this problem. Reworking node startup and initialisation cut machine boot time by 83%, taking it from two-to-seven minutes down to under thirty seconds, and the follow-on work on container image distribution and memory snapshots took the user-visible cold start down to seconds. The point is not that we solved it. The point is that this is where the money is, and a general-purpose instance API does not expose the levers.
The case for staying on AWS, made honestly
Any comparison that concludes "leave AWS" without examining AWS's own AI silicon is not worth reading. Inferentia 2 and Trainium are genuinely good, and for a large class of workloads they are the correct answer.
We benchmarked them. For many ML use cases we found Inferentia 2 to be up to 50% cheaper than similar AI chips at the same performance. Against an Ampere A10 baseline of roughly 600 tokens per second at $0.000555 per second, Inferentia generated 1014 tokens at $0.000479 per second — more throughput for less money. On latency it was better too: 28.57ms per token versus 41.77ms for TensorRT.
Then it inverts, and this is the part most comparisons omit. On a Llama 3 70B model, Inferentia ran at 223 tokens per second at $0.00271 per second, against TensorRT's 663 tokens per second at $0.00358 per second across two 80GB GPUs. That is roughly a third of the throughput for about 25% less cost. Whether that is a good trade depends entirely on whether you are latency-bound or throughput-bound, and nobody can answer that for you.
The catch is portability. Inferentia requires compiling through the Neuron SDK, so the performance is real but the lock-in is real as well. Our full Trn1/Inf2 benchmark has the complete methodology.
The three kinds of alternative
Once you accept that provisioning is the axis, the options sort into three groups rather than a flat list of vendors.
| Category | What you get | What it costs you | Best when |
|---|---|---|---|
| Hyperscaler AI silicon | Inferentia, Trainium, TPUs. Real price-performance gains | Compilation toolchains and vendor-specific model formats | You have one or two stable models and volume to justify porting |
| Serverless GPU platforms | Scale-to-zero, per-second billing, seconds-long cold starts | Less control over the machine; you work through the platform's abstractions | Traffic is spiky, or you deploy often |
| GPU marketplaces | The lowest raw hourly rates available | You operate it: orchestration, autoscaling, reliability | You have platform engineers and steady, predictable load |
Most teams comparing AWS alternatives for AI workloads are implicitly asking about the middle row, because the first requires a porting budget and the third requires a platform team. Our wider comparison of alternatives to AWS, GCP and Azure covers the vendor-by-vendor version, and the serverless GPU provider comparison has the platform-level detail.
What actually changes when you move
Cold start is the number to compare, because it decides whether scale-to-zero is available to you. In our provider comparison, Cerebrium and Beam led at 2–4 second startup times, RunPod and Google Cloud Run landed mid-range, and Baseten trailed at 16–60 seconds. With 2–4 second cold starts, workloads spin up only when requests arrive and shut down immediately afterwards — which is the mechanism, not the marketing.
Two customer migrations give a sense of the magnitude. bitHuman moved off a hyperscaler and saw launch times fall from 30 seconds to under 10, and under 3 seconds when warm — their full migration is written up here. Distil Labs cut inference costs by 50% while accuracy went from 83% to 92%, holding latency and reliability steady at production scale. Neither number came from a cheaper GPU hour. Both came from paying for less idle time.
How to actually evaluate this
Five questions, in the order that matters:
What is the cold start, measured on your image? Not the vendor's number on a hello-world container. Your model weights, your dependencies.
Can you scale to zero, and what does the first request after a trough cost the user? If the answer is "we keep one warm", price that instance and add it to the bill.
What is your average concurrency, not your peak? Per-request cost is instance cost divided by average concurrency, and average is always well below peak on real traffic.
What does porting cost? Neuron compilation, custom containers, and framework constraints are real engineering time. Count it.
Where are your users? Region coverage moves latency far more than chip choice for anything interactive.
If cost pressure is the immediate driver, cutting AI infrastructure costs without giving up performance is the broader version of this argument, and the top serverless GPU providers is the shortlist version. Current per-second rates for the GPU classes referenced here are on the pricing page — we have deliberately not quoted list prices in this article, because they change and a stale price is worse than no price.
The short version
Leave AWS if your traffic is spiky and you are paying for idle GPUs to hide slow provisioning. Stay on AWS silicon if you have one or two stable, high-volume models and the engineering budget to compile for Inferentia. Use a marketplace if you have a platform team and flat load.
What you should not do is pick on hourly rate. It is the least decision-relevant number in the comparison, and it is the only one most comparisons show you.
Frequently asked questions
- What are the best AWS alternatives for AI workloads?
- They fall into three groups. Hyperscaler AI silicon such as Inferentia and Trainium gives real price-performance gains but requires compiling through vendor toolchains. Serverless GPU platforms such as Cerebrium, Beam, Modal and Baseten give scale-to-zero and per-second billing. GPU marketplaces such as RunPod give the lowest raw hourly rates but you operate the orchestration yourself. Which is best depends on how spiky your traffic is and whether you have a platform team.
- Is moving off AWS actually cheaper for inference?
- Usually, but not because the GPU hour is cheaper. The saving comes from no longer paying for idle capacity kept warm to hide slow provisioning. Distil Labs cut inference costs by 50% on Cerebrium while raising accuracy from 83% to 92%, and bitHuman saw launch times drop from 30 seconds to under 10. Both gains came from utilisation, not list price.
- Is AWS Inferentia good for AI inference?
- For many workloads, yes. In our benchmark Inferentia 2 was up to 50% cheaper than similar AI chips at the same performance, generating 1014 tokens at $0.000479 per second against an A10 baseline of about 600 tokens per second at $0.000555 per second, with lower per-token latency. On larger models the trade shifts: Llama 3 70B ran at 223 tokens per second on Inferentia versus 663 on TensorRT, for roughly 25% less cost. The constraint is that you must compile through the Neuron SDK.
- Why is cold start the number to compare?
- Because it decides whether scale-to-zero is available. If provisioning takes minutes you must keep GPUs warm through every traffic trough, and idle GPU time is typically the largest avoidable cost in an inference budget. With 2-4 second cold starts, workloads spin up when requests arrive and shut down immediately after.
- How slow is EC2 provisioning for AI workloads?
- On our own AWS-backed fleet, the time from requesting a machine to it being ready to run user containers was two to seven minutes, largely because booting required loading more than 17 GB of platform container images. After reworking node startup and image distribution we brought that to under 30 seconds. The instance API itself does not expose the levers that matter here.
Sources
- cerebrium.ai
“From the moment we requested a machine to the moment it was ready to run user containers, the process could take anywhere from 2 to 7 minutes.”
Time from requesting an EC2 machine to it being ready to run user containers, measured on our own AWS-backed fleet. This is the provisioning unit teams inherit by default.
- cerebrium.ai
“The issue was that booting a machine required loading more than 17 GB of platform container images.”
Volume of platform container images that had to be loaded on every machine boot, the mechanical reason general-purpose instance startup is slow for AI images.
- cerebrium.ai
“The result was a dramatic improvement: we reduced machine boot time from 2–7 minutes to under 30 seconds.”
Result after reworking node startup and image distribution: the same AWS hardware, a different provisioning path.
- cerebrium.ai
“In this post, we show how we reworked node startup and initialization on AWS to reduce machine boot time by 83%, cut the long tail of cold starts, and lower the amount of excess capacity we needed to keep running, improving overall utilization.”
Scope of the node-startup rework and what it bought in excess-capacity terms.
- cerebrium.ai
“For many ML use cases, we have seen Inferentia 2 be up to 50% cheaper than similar AI chips while giving the same performance.”
Price-performance of AWS's own inference silicon versus comparable AI chips, from our benchmark of Inferentia 2.
- cerebrium.ai
“an Ampere A10 had a throughput of ~600 tokens per second (FP16) at a cost of $0.000555 per second.”
Ampere A10 baseline in the same benchmark: throughput and cost per second.
- cerebrium.ai
“Inferentia are able to generate 1014 tokens at a cost of $0.000479 per second.”
Inferentia 2 in the same benchmark: throughput and cost per second, directly comparable to the A10 baseline above.
- cerebrium.ai
“Additionally, Inferentia generates a per token latency of 28.57ms per token whereas TensorRT was 41.77ms per second and therefore is better for latency sensitive applications.”
Per-token latency of Inferentia 2 versus TensorRT in the same benchmark.
- cerebrium.ai
“Comparatively, you can run a Llama 3 70b model at 223 tokens per second at a cost of $0.00271 per second vs using TensorRT which has a throughput of 663 across 2 80GB GPUs at a cost of $0.00358 per second.”
The Llama 3 70B case, where Inferentia's advantage inverts: about a third of the throughput for roughly 25% less cost.
- cerebrium.ai
“Cerebrium and Beam lead in cold start performance, both achieving 2–4 second startup times, while RunPod and Google Cloud Run fall in the mid-range, and Baseten trails with 16–60 second startup delays.”
Cross-provider cold-start comparison, which determines whether you can scale to zero between traffic peaks.
- cerebrium.ai
“The turning point came when one of their engineers ran an internal demo with Cerebrium—launch times plummeted from 30 seconds to under 10, and even under 3 seconds when warm.”
bitHuman's measured launch-time change after moving off a hyperscaler.
- cerebrium.ai
“With Cerebrium, we reduced their inference costs by 50%, and increased accuracy from 83% to 92%, while keeping latency and reliability consistent at production scale.”
Distil Labs' measured inference-cost and accuracy change on Cerebrium.
- cerebrium.ai
“With 2-4 cold starts, your workloads spin up only when requests arrive and shut down immediately after (configurable by you).”
What a 2-4 second cold start permits operationally: scale-to-zero between requests.
