Managing LLM Model Versions: Staging to Production

Managing model versions across staging and production for LLM inference is less about tracking weights and more about controlling a release boundary. A served LLM behaves differently if you change its tokenizer, its generation config, its serving code, or its inference engine, so the unit you promote has to be the whole bundle, not a checkpoint. This guide is an operator's playbook: build each version as an immutable artifact, promote it through shadow then canary stages gated on quality signals, and keep rollback down to repointing an alias.

Why a model version is more than the weights

The most common LLM-serving mistake is versioning only the weights. As the LLM Serving guide puts it, "The single most common LLM-serving mistake is versioning only the weights. A served model is a bundle. Change any component and outputs can move. Pin all of it or you have not pinned anything." That bundle includes model config, tokenizer, generation config, serving and adapter code, the inference engine version, the quantization recipe, and runtime dependencies.

For an AI application the boundary is even wider. A production release is a release manifest - model, prompt, retrieval, and runtime configuration tested and promoted together. Stack Overflow's engineering blog frames it directly: "For an AI application, a model version is only part of the answer. Inputs, preprocessing, prompts, retrieval, tool contracts, and serving settings can change behavior independently." Pin the manifest, not the file. The exact engine flags you pin matter too; our vLLM server parameter notes show how much throughput shifts when those settings change.

Model registry: immutable identity vs mutable aliases

A model registry is the operational backbone here. Snowflake defines it as "a system for managing identifiable versions of machine learning models and the metadata and relationships needed to operate them through their lifecycle." Each entry "typically includes the model artifact, evaluation metrics, input and output signature, training lineage, ownership, aliases and information about its lifecycle or deployment state."

The critical design choice is separating two layers. Good versioning keeps immutable identity apart from mutable pointers: "Immutable identity - a content hash or an append-only version number that never moves... Mutable pointers (aliases/stages) - human-friendly names like @champion, @production, staging that point at an immutable version and can be repointed during a promotion or rollback." You audit against the hash; you route against the alias. MLflow's registry, for example, "automatically tracks versions of each model, allowing teams to compare iterations, roll back to previous states, and manage multiple versions in parallel (e.g., staging vs. production)."

Promoting staging to production safely

Do not cut traffic over in one step. Start in shadow mode: "duplicate production requests to both the current model (which serves users) and the candidate model (which doesn't). Log both outputs, compare them, and make a promotion decision based on what you observe." You can even front-run deployment by replaying history - running "shadow mode agents on historical production requests before you deploy anything" gives a fast read on regression areas before you touch production infrastructure.

Once shadow mode is clean, move to canary. Route a small share - "start at 1%, sometimes as low as 0.1% for high-stakes applications" - then "gradually increase the canary's traffic share: 1% → 5% → 20% → 50% → 100%" while metrics stay in bounds. The mechanics of wiring this on a GPU endpoint are covered in depth in our canary rollout and rollback walkthrough.

The gate matters as much as the ramp. For LLMs the signal set is broader than errors and latency: a new version "can be technically flawless, with zero errors and lower latency, yet produce subtly worse reasoning, shift tone, or hallucinate more frequently." So monitor error and 5xx counts, P99 latency and GPU OOM events, automated quality scores for hallucination and coherence, LLM-as-a-Judge results, and business signals like task completion and user satisfaction. Pin those gates to the latency budget you already run - see our notes on goodput and latency SLOs.

Rollback with stable, candidate, and previous aliases

Rollback is where immutability pays off. The foundation is that "every model artifact is immutable, content-addressed, and tagged before it touches production," with the registry maintaining three production aliases: stable (currently serving, proven), candidate (under canary or shadow), and previous. Because the bundle is pinned, recovery is one operation - repoint stable at the previous version - with no rebuild and no ambiguity about what you reverted to. The step-by-step rollback flow for a live endpoint is in the same canary and rollback guide.

Runtime model and prompt management

If changing a prompt or swapping a model requires a redeploy, every one of the steps above becomes slow and risky. Externalize model selection and prompts from code: "The ability to update prompts, swap models, or adjust inference parameters without shipping new code is the foundation on which everything else depends." Serve behind a stable contract so the alias swap is invisible to callers - an OpenAI-compatible endpoint lets you move versions underneath without touching client code.

Infrastructure considerations

Shadow and canary both mean running two versions at once, so you need headroom for parallel deployments. In our testing, Cerebrium cold starts land in the 2-4 second range, which keeps a freshly-promoted candidate from adding a long warm-up tax when canary traffic first hits it. For steady dual-version serving you want predictable headroom - our guide to reserved GPU capacity for production LLM inference covers sizing that baseline. Where versions differ only by adapter, you can often avoid doubling hardware by co-locating multiple models on one GPU.

Fine-tuned variants are a version-management problem of their own: each adapter is a distinct entry in the registry with its own bundle and aliases. Our guide to serving fine-tuned LLMs on serverless GPUs walks through keeping those straight. The cost of carrying parallel versions is real but bounded; for reference, our serverless GPU cost breakdown prices a 2xH100 configuration at $0.11604 per minute.

Reference staging-to-production runbook

1. Build the full bundle (weights, tokenizer, generation config, serving code, engine version, quantization recipe, dependencies) and register it with a content hash.

2. Replay last week's production traffic through the candidate; have a judge compare against current outputs.

3. Shadow live traffic to the candidate; serve nothing, log everything, compare.

4. Canary 1% → 5% → 20% → 50% → 100%, holding at each step until the full signal set stays in bounds.

5. Promote by repointing the stable alias; keep the prior version tagged previous.

6. Rollback instantly by repointing stable back to previous if any gate fails.

The short version: pin the whole bundle, route with aliases, gate every promotion on quality signals, and make rollback a one-command alias swap. Memory-snapshot cold starts, covered in our GPU cold start deep dive, keep that swap fast enough to treat as routine.