Multiple Models on One GPU: Pack or Dedicate?

TL;DR

There are two ways to size an inference fleet. GPU-per-model gives every model its own accelerator: simple, isolated, and the safe default. Model-packing (multiplexing) puts several models on one GPU and swaps between them, reclaiming the capacity a dedicated GPU leaves idle. The safe default has a hidden tax: resource fragmentation. If your models are small, bursty, or numerous, packing usually wins on cost. If a single model saturates a GPU on its own at steady load, dedicate. This guide gives you the decision framework and the proof points behind it.

The two architectures

GPU-per-model is one deployment, one model, one (or more) whole GPUs. Each model scales independently, failures are isolated, and there is nothing to reason about beyond the model itself. It is the pattern almost every team starts with, and for good reason.

Model-packing / multiplexing loads multiple models into one GPU's memory and time-slices the compute between them. When a request arrives for model B while model A is loaded, the stack swaps context and serves it. IonRouter describes exactly this approach: "Our custom inference stack multiplexes models on a single GPU, swaps in ms, and adapts to traffic in real time." The unit of billing stops being the model and becomes the GPU-second, which is what changes the economics.

Why GPU-per-model wastes money: resource fragmentation

The problem with one-GPU-per-model is not the GPU you use, it is the GPU you reserve but don't use. Real traffic is bursty. A model provisioned for peak sits mostly idle at the trough, and its GPU cannot be lent to a busier neighbour because it belongs to a different deployment. Multiply that across a dozen models and you are paying for a fleet that is, on average, half-empty.

Anyscale names this precisely: "Resource fragmentation occurs when scaling activities lead to uneven resource utilization across nodes." Every scale-up and scale-down leaves stranded capacity behind. The dedicated-GPU model doesn't overspend on any one request; it overspends on the headroom it can never reclaim. If you are trying to bring down your LLM inference cost at scale, fragmentation is usually the first place to look.

How model-packing reclaims the GPU

Packing works when swapping between models is cheap enough that idle headroom disappears. The bar is sub-second, and increasingly sub-millisecond. IonRouter reports model "swaps in ms" and, in a production case, "Five vision-language models on a single GPU - 2,700 video clips, concurrent users, <1s cold starts." Five models where a dedicated fleet would have provisioned five GPUs.

The enabling technology is fast loading and fast restore. Slow cold starts are what make packing feel risky, because every swap looks like a fresh boot. That is exactly the tax memory snapshots attack. In our benchmarks, Cerebrium snapshots reduced cold starts by an average of 71% versus running the same workloads without snapshots, with reductions as high as 88% on vLLM - the full method and curves are in our write-up on reducing GPU cold starts with memory snapshots. When restoring a model is near-instant, keeping many models on one GPU stops costing you latency. We go deeper on the mechanism in our write-up on serverless GPU cold starts, and on the node-boot side we reduced machine boot time by 83% by reworking how platform images load.

Does packing cost throughput?

The fear is that sharing a GPU throttles each model. In practice a well-engineered stack can push a shared GPU harder than a naive dedicated one. Modal's Quail is the extreme proof: "On one multi-join query where planning is particularly important, Quail hits over a billion tokens processed per minute per H100 GPU (TPM/GPU), >10x faster than our vLLM baseline on the same hardware." A single H100, saturated, does not need a second GPU to be fast - it needs to not sit idle.

The lesson generalises. Throughput is lost to underutilization, not to sharing. If your models rarely max out the card, the throughput they leave on the table is exactly what a co-tenant reclaims. This is closely related to how H100 vs H200 inference throughput plays out: the ceiling only matters if you can reach it.

Operating a multi-model GPU in production

Packing is an operations discipline, not just an allocation trick. Two capabilities make it safe.

Heterogeneous co-location. Ray Serve lets models of different sizes share a service: "Ray Serve makes it possible to create deployments for Llama-3-8B and Llama-3-70B on the same Service with different resource requirements (1 GPU and 4 GPU per replica respectively)." You are not forced to pack identical models.

Automatic defragmentation. As replicas scale up and down, they scatter. Anyscale's Replica Compaction addresses this directly: "With Replica Compaction, Anyscale will automatically migrate replicas into fewer nodes in order to optimize resource use and reduce costs." Consolidation is what turns theoretical density into a smaller bill.

Roll new models onto a shared GPU the same way you would any endpoint - see our guide to canary rollout and rollback on a serverless GPU endpoint. Routing matters too: token-load-aware routing keeps a busy co-tenant from starving its neighbours. Cerebrium removes much of this operational drag; in one migration we describe cutting GPU costs nearly in half while reducing engineering overhead by more than 90%.

Decision framework: pack or dedicate

Rules of thumb. Pack when several models each use a fraction of a GPU and their peaks don't coincide - the classic fleet of small vision, embedding, or fine-tuned models. Dedicate when a single model runs hot enough to keep a GPU busy on its own, or when tenancy isolation is a hard requirement. When you are between the two, packing plus fast swaps and automatic compaction is the lower-risk bet, because you can always split a hot model back out. For fine-tuned variants specifically, see serving fine-tuned LLMs on serverless GPU; for the smaller end of the fleet, concurrent voice sessions per GPU shows how far density can go.

The cheapest GPU is the one you already own, fully used. Cerebrium's serverless GPU platform is built to keep it that way, with 2-4 second cold starts and per-second billing described in our AWS alternatives guide.