Serving Fine-Tuned LLMs on Serverless GPUs

Fine-tuning is the easy half. The moment you have your own weights you leave the world of per-token API pricing and enter the world of capacity planning, and nothing about the training run prepares you for it. Serving fine-tuned llms on serverless gpu infrastructure raises exactly three questions that a hosted endpoint never asks you: how long do the weights take to load, which card do they fit on, and what happens when you have twenty of them instead of one.

Does a fine-tuned model need its own GPU?

Usually yes, and usually a smaller one than you expect. That is the whole economic case for fine-tuning: a task-specific model does one job at a size a general model cannot match.

The production example we measured is Distil Labs, who replaced large general models with small tuned ones and cut inference costs by 50% while accuracy rose from 83% to 92%. The accuracy number is the important one. Cost savings from a smaller model are only interesting if quality survives, and here it improved.

The size argument shows up directly in throughput. A Llama-3-8B served with TensorRT-LLM reaches around 1700 output tokens per second in FP8 on a single A10. Put the same workload on an H100 and you have bought headroom you are not using. Our published rates put the A100 80GB at $0.000592 per GPU-second against the H100 at $0.000953, so right-sizing is a real saving rather than a rounding error. Choosing the engine is a separate decision, and our vLLM, SGLang and TensorRT-LLM benchmark covers that comparison.

The weight-load tax, and how to stop paying it

Here is the cost a hosted API genuinely does hide. Your checkpoint has to reach GPU memory before it can answer anything.

Loading a model into GPU memory takes roughly 10 to 15 seconds on every start, and a full vLLM start is far worse. Snapshotting the process after the weights are resident changes the shape of the problem entirely: restoring from a 9 GiB checkpoint reduced startup to 2.25 seconds from S3.

Why that matters more for fine-tuned models than for anything else: you have many of them, each serving a slice of traffic, and each idle for most of the day. If start-up is 50 seconds you keep every one warm and pay for idle GPUs. If it is two seconds you scale each to zero, and new workers provision in 2 to 4 seconds when traffic returns.

Many small models, not one big one

This is the structural difference, and it is the one most serving platforms handle badly.

Distil Labs runs up to 150 requests per second per model during high-traffic periods, with multiple models deployed concurrently. That is not one endpoint under load. It is a fleet of endpoints, each with its own traffic shape, its own scaling curve and its own idle period.

Three consequences worth planning for:

1. Per-model autoscaling, not per-cluster. A shared pool sized for the busiest model wastes capacity on all the others.

2. Deployment speed becomes an iteration cost. You redeploy after every training run, so build times of 8 to 14 seconds are the difference between a tight loop and a coffee break.

3. Per-second billing does the accounting. Twenty models each busy 5% of the day is one GPU's worth of work spread over twenty endpoints, and only per-second pricing bills it that way.

The general version of this deployment problem is in our guide to deploying machine learning models, and if you are serving an open-weight model rather than your own tune, our DeepSeek-R1 walkthrough is the closest analogue.

Work out your own number

1. Measure weight-load time on your actual checkpoint, not the base model. Adapters merged into the weights change it.

2. Find the smallest card the model fits on at your batch size and precision, then price that card, not the one you trained on.

3. Multiply by realistic utilisation per model. With many endpoints, per-model idle time is the dominant term.

4. Decide warm or cold per model, using measured restore time rather than a platform default.

The full arithmetic for step 3 is in our tokens-per-minute cost breakdown. Current rates are on the pricing page; the figures here come from our published benchmarks and will drift, so re-derive before you budget against them.

The short version

Serving your own checkpoint costs weight-load time, a card decision and per-model idle. Snapshot restore took startup from about 50 seconds to 2.25 in our measurements, right-sizing from an H100 to an A100 is a 38% rate difference, and a fleet of small tuned models held 150 requests per second each while cutting cost in half against the general model they replaced.