Loading Model Weights from Object Storage at Deploy
Connor Blier
Founding GTM
Loading Model Weights from Object Storage at Deploy
Loading large model weights from object storage at deploy time means pulling tens to hundreds of gigabytes of checkpoints from S3, GCS, or Azure Blob onto the GPU during a cold start. To cut that time, stream tensors directly to GPU memory instead of staging the full file on local disk first.
Loading large model weights from object storage at deploy time is the slowest, most expensive step in a GPU cold start. For frontier-scale models, weights alone can be 100s of GB, and moving them from a bucket to the GPU is where most of your startup budget disappears. This guide decodes that path phase by phase and shows the three patterns for getting weights onto the GPU: bake into the image, pull at startup, or stream directly.
We have gone deep on the related voice-AI and 70B autoscaling angles elsewhere; this piece focuses on the object-storage-to-GPU critical path. For how these same phases play out when you scale a 70B replica, see our write-up on autoscaling Llama 70B replica startup.
The cold-start path, phase by phase
A cold start is a sequence of hops, and weight loading is the tail that dominates. The phases are container image download, layer extraction, startup configuration, and loading weights into GPU memory. Standard loading tends to run these phases serially rather than overlapping them, so the download, extraction, and disk-writing steps wait on each other instead of progressing in parallel.
Even reading from fast storage, a standard Hugging Face load can take 40+ seconds at ~2 GB/s, and for frontier-scale models the weight transfer is far larger still. The fix is to attack the weight path directly: change the format, change the placement, and change the loader.
Object storage options: S3, GCS, Azure
Your weights live in object storage, and the loader has to bridge that to the GPU. Loading from cloud storage such as Amazon S3 involves an extra step: weights are first downloaded to local disk before being moved into CPU and then GPU memory. Modern streaming loaders remove that detour by pulling tensors straight from the bucket, and most support the major object stores (S3, GCS, and Azure Blob) through S3-compatible APIs.
Three patterns to get weights onto the GPU
There are three ways to place weights, and the choice is the spine of your cold-start design.
Bake into the image. Simplest, but the weights inflate your image. As Cerebrium's docs note, baking weights into the container increases image size, which means the platform has to pull and restore a larger image before your application can start initialization. Moving weights to persistent storage shrinks the image so the pull finishes sooner.
Pull at startup. Download weights from object storage when the container boots. Critical detail: run initialization at module scope, not per request, so weights hit the GPU once per container instead of once per call. Pair it with a cache volume and the model downloads from Hugging Face exactly once, on the first container that ever runs.
Stream direct to GPU. Skip the full local-disk stage and stream tensors straight into GPU memory. For large models this avoids the serial remote-to-disk-to-GPU hops that dominate standard loading, and it is the focus of the rest of this guide.
Streaming, lazy loading, and why safetensors
Streaming only works if your checkpoint format allows it, which is why safetensors is a prerequisite. The safetensors format stores tensors in a flat binary layout with a small index header, so the loader can mmap specific shards and overlap disk reads of shard N+1 with CUDA transfers of shard N. That overlap is the whole game. Safetensors is also now mainstream, with 42% of Hugging Face models (621,008 of 1,492,435) tagged safetensors as of March 2025.
Streaming loaders compared
Several loaders implement this streaming pattern, each with its own benchmarks.
Run:ai Model Streamer concurrently reads weights and streams them into GPU memory. On AWS g5.12xlarge with Llama 3 8B (15 GB), it loaded the model in 4.88 seconds on Amazon S3 at concurrency 32. Its sharded loader is efficient for tensor-parallel models because each worker only needs to read its own shard rather than the entire checkpoint.
Tensorizer uses zero-copy loading, pulling tensors chunk by chunk from storage directly to GPU via a bespoke format with metadata at the file beginning. CoreWeave reports it delivered >5x faster latency per request versus Hugging Face when scaling from zero.
fastsafetensors improves safetensors deserialization, delivering 4.8x to 7.5x speedups across Llama, Falcon, and Bloom models.
FlashPack flattens model state into a contiguous stream and pipelines I/O, achieving 3-6x faster loading than accelerate or the standard load_state_dict() flow.
If you are tuning the server itself, our vLLM server parameters guide covers the knobs that pair with these loaders.
GPU Direct Storage
GPU Direct Storage (GDS) removes the CPU from the data path. GDS enables a direct DMA path between GPU memory and storage, avoiding a bounce buffer through the CPU. For deployments reading very large checkpoints off local NVMe, this eliminates the CPU-staged copies that otherwise cap throughput.
Caching and persistence
The download you never repeat is the one that costs nothing. Persisting weights on a cache volume means subsequent cold starts read the checkpoint from the volume instead of pulling it across the network again. At the platform level, multi-tier caching co-located with workloads plus single-flight downloads that prevent thundering herd issues deliver 2-3x faster cold starts during burst scaling. Memory snapshots are another lever; see our work on reducing GPU cold starts with memory snapshots.
Reference architectures
The proven pattern combines all of the above: store weights in safetensors in object storage, shrink the container image, stream with a concurrent loader, and cache on a persistent volume. BentoML reports it achieved 25x faster cold starts, with true on-demand model loading. The platform itself matters: in our benchmarks, Cerebrium's own boot-time work reduced machine boot time from 2-7 minutes to under 30 seconds. For teams weighing providers, our guides on AWS alternatives for AI workloads, CoreWeave alternatives for production LLM inference, and the broader alternatives to AWS, GCP and Azure for deploying AI models compare the trade-offs, and LLM inference cost at scale connects cold starts to the bill.
Decision checklist
Convert checkpoints to safetensors so streaming and lazy loading are even possible.
Do not bake multi-GB weights into the image; keep them in object storage or on a volume.
Choose a streaming loader (Run:ai Model Streamer, Tensorizer, fastsafetensors, FlashPack) and tune concurrency for your bucket.
Use sharded loading so each tensor-parallel worker reads only its shard.
Enable GDS for NVMe-backed reads of very large checkpoints.
Cache weights on persistent storage so repeat cold starts skip the network.
Initialize at module scope, once per container, not per request.
For model-serving design beyond cold starts, see our guide on serving fine-tuned LLMs on serverless GPUs.
Frequently asked questions
- Why is loading weights the slowest part of a cold start?
- For large models, weight download from storage dominates. Standard loading moves data through several serial hops before the model reaches the GPU, and a plain Hugging Face load can take 40+ seconds even reading at ~2 GB/s.
- Do I need safetensors to stream weights?
- Effectively yes. Safetensors stores tensors in a flat binary layout with a small index header, so a loader can mmap specific shards and overlap the disk read of shard N+1 with the CUDA transfer of shard N.
- How do streaming loaders compare?
- It depends on your stack and benchmarks vary. Run:ai Model Streamer loaded a 15 GB Llama 3 8B in 4.88 seconds from S3 at concurrency 32; Tensorizer reported >5x faster latency per request than Hugging Face when scaling from zero; fastsafetensors reports 4.8x to 7.5x speedups; and FlashPack reports 3-6x over standard loading.
- What does GPU Direct Storage change?
- GDS enables a direct DMA path between GPU memory and storage, avoiding a CPU bounce buffer, which eliminates intermediate CPU memory copies for large checkpoint reads.
- Should I bake weights into the container image?
- Generally no. Baking weights increases image size, so the platform must pull and restore a larger image before initialization starts. Keeping weights in object storage or on a persistent volume shrinks the image and lets weight loading begin sooner.
Get started with Cerebrium
Deploy AI models on serverless GPUs in minutes, with no infrastructure to manage. Start for free and pay only for the compute you use.
Sources
- bentoml.com
“1. **Download:** Pull all image layers from the registry. 2. **Extract**: Decompress layers and write them to disk. 3. **Startup**: Set up mounts, network, GPU access, and other configurations. 4. **Load model**: Load weights into GPU memory.”
Cold-start phase breakdown.
- baseten.co
“**Weight download:** This is where things can get expensive, especially for large models. Your model weights have to be transferred from wherever they're stored to the node's local disk. For frontier-scale models, this can easily be 100s of GB of weights.”
Weight download dominates cold start.
- developer.nvidia.com
“Loading models from cloud-based storage such as Amazon S3 often involves an extra step: the weights are first downloaded to local disk before being moved into CPU and then GPU memory.”
S3 extra local-disk step.
- spheron.network
“safetensors format breaks this by storing tensors in a flat binary layout with a small index header. The loader can mmap specific shards directly and overlap disk reads of shard N+1 with CUDA transfers of shard N.”
Safetensors enables overlap.
- arxiv.org
“As of March 2025, 621,008 out of 1,492,435 models (42%) in the Hugging Face model repositories are tagged as "Safetensors"”
Safetensors adoption.
- developer.nvidia.com
“Benchmarks on AWS g5.12xlarge instances with a Llama 3 8B model show Model Streamer loading the 15 GB model in 14.34 seconds on GP3 SSD at concurrency 16, 7.53 seconds on IO2 SSD at concurrency 8, and 4.88 seconds on Amazon S3 at concurrency 32.”
Run:ai Model Streamer S3 benchmark.
- docs.vllm.ai
“The sharded loader is particularly efficient for tensor or pipeline parallel models where each worker only needs to read its own shard rather than the entire checkpoint.”
Sharded loader efficiency.
- coreweave.com
“Instead of loading the whole model into RAM before transferring it to the GPU, Tensorizer pulls it over chunk by chunk. This "tensor streaming" process is enabled by Tensorizer's bespoke serialization format that puts all the necessary metadata at the beginning of a single binary file.”
Tensorizer zero-copy streaming.
- coreweave.com
“The average latency per request was >5x faster for Tensorizer compared to Hugging Face when scaling from zero, and required fewer pod spin ups and less RAM.”
Tensorizer vs HF scaling from zero.
- arxiv.org
“Experimental results show performance improvements of 4.8x to 7.5x in loading models such as Llama (7, 13, and 70 billion parameters), Falcon (40 billion parameters), and the Bloom (176 billion parameters).”
fastsafetensors speedup.
- blog.fal.ai
“With FlashPack, loading any model can be 3–6× faster than with the current state-of-the-art methods like `accelerate` or the standard `load_state_dict()` and `to()` flow — all wrapped in a lightweight, pure-Python package that works anywhere.”
FlashPack 3-6x.
- lmsysorg.mintlify.app
“When loading models from object storage, SGLang uses a two-phase approach: 1. **Metadata Download** (once, before process launch): Configuration files and tokenizer files are downloaded to a local cache 2. **Weight Streaming** (lazy, during model loading): Model weights are streamed directly from object storage as needed”
SGLang two-phase loading.
- docs.nvidia.com
“GDS enables a direct data path for direct memory access (DMA) transfers between GPU memory and storage, which avoids a bounce buffer through the CPU.”
GDS direct DMA.
- cerebrium.ai
“For large models, a standard Hugging Face load can take 40+ seconds even when reading from storage at ~2 GB/s.”
Standard HF load time.
- cerebrium.ai
“Baking weights into the container increases image size, which means Cerebrium has to pull and restore a larger image before your application can start initialization.”
Baking weights inflates image.
- beam.cloud
“An `on_start` function runs when the container boots and its return value is handed to every request through `context.on_start_value`, so the weights hit the GPU once per container instead of once per call.”
Module-scope init.
- beam.cloud
“Point `cache_dir` at a mounted `Volume` and the model downloads from Hugging Face exactly once, on the first container that ever runs. After that every cold start reads the checkpoint from the volume instead of pulling it across the network again.”
Volume caching.
- bentoml.com
“Ultimately, we developed a solution that achieved 25x faster cold starts, with true on-demand model loading”
BentoML 25x.
- baseten.co
“BDN delivers 2–3x faster cold starts through multi-tier caching and single-flight downloads that prevent thundering herd issues during burst scaling.”
Baseten multi-tier caching.
- cerebrium.ai
“The result was a dramatic improvement: we reduced machine boot time from 2–7 minutes to under 30 seconds.”
Cerebrium boot-time improvement.
