Canary Rollout & Rollback on Serverless GPU Endpoints

Canary rollout and rollback for a serverless gpu inference endpoint borrows a playbook that was designed for always-on servers, where every revision is already running and a fallback is just a router entry. On serverless GPU, two of those assumptions quietly break. This guide adapts the standard canary/rollback flow to the reality that a new revision has to cold-start before it can take traffic, and that the safe rollback target is not free to keep warm.

The short version: rollback is a routing switch to a pre-warmed prior revision, not a redeploy. If you have to redeploy and cold-start to roll back, you have already failed your incident.

What canary and rollback normally assume (and why serverless GPU breaks it)

The classic model is well specified. KServe supports canary rollouts that let a new version receive a percentage of traffic rather than an all-or-nothing cutover, and "KServe will then automatically split the traffic between the last good revision and the revision that is currently being rolled out according to the canaryTrafficPercent value." For safety it "automatically tracks the last good revision that was rolled out with 100% traffic," and if something goes wrong "100% of the traffic will be pinned to the previous healthy/good revision- the PreviousRolledoutRevision." Notably, this strategy "is only supported in serverless deployment mode" in KServe.

Two assumptions underneath that model do not hold on a scale-to-zero GPU endpoint:

• The new revision is not warm. On an always-on cluster the canary target is already running. On serverless it may be scaled to zero. Even best-in-class startup is not instant: Cerebrium's own serverless inference runs with "cold start times of 2-4 seconds," which means the first canary requests hit a revision that is still coming up unless you warm it first.

• The fallback is not free. Pinning traffic back to PreviousRolledoutRevision only works instantly if that revision is still warm. Keeping a GPU revision warm means paying for an accelerator that is serving little or no traffic. That is the cost this guide exists to bound.

Prerequisite: getting a new revision ready to take traffic

Before a revision is canary-eligible it must be loaded and warm. Inference servers separate loading from serving for exactly this reason. NVIDIA Triton "operates in one of three model control modes: NONE, EXPLICIT, or POLL," and the explicit mode lets you load a model into a server without immediately routing production traffic to it. Use that window to pull the image, initialize CUDA, load weights, and run a smoke request.

Treat cold-start as a gate: the revision is not allowed into the traffic split until it has completed a warm-up request end to end. Container distribution dominates that number, and it is worth engineering. In Cerebrium's own container-distribution benchmark, "a traditional pull must download and unpack the entire 4.87GB image, while the lazy-loading path only downloads a ~15MB metadata bootstrap before the container can start." The faster the revision reaches warm, the cheaper the whole canary window is. If you are moving between platforms, reproduce your real cold start first, as covered in our Modal alternatives guide.

Running the canary

Once the new revision is warm, assign it a slice of traffic. In KServe "the canaryTrafficPercent field in the component's spec needs to be set with the percentage of traffic that should be routed to the new revision," and the router handles the split against the last good revision. Start small, watch metric gates (error rate, p95 latency, token latency for LLMs), and ramp only when they hold.

The serverless-specific rule: keep the previous revision warm for the entire canary window. It is your rollback target, and it is only instant while it is warm. This is the deliberate cost. A useful reference point for latency gates is that inference hardware choices move real numbers; in Cerebrium's own Llama 3 8B comparison, "Inferentia generates a per token latency of 28.57ms per token whereas TensorRT was 41.77ms." Gate on the metric that matters to your users, not just on HTTP success. For cost gating during the canary, see our notes on LLM inference cost at scale.

KServe also protects you here: "if there is an unhealthy or bad revision applied, traffic will not be routed to that bad revision."

Rolling back safely

Rollback has two moves. First, route 100% of traffic back to the pinned prior revision. Because you kept it warm, this is a pure routing change with no cold-start penalty. Second, unload the failed revision so you stop paying for it. Triton's explicit control mode lets you unload a model on demand; on serverless GPU, unloading is also how you stop the meter. Leaving a broken revision loaded burns GPU-seconds for nothing.

The order matters: switch traffic first, unload second. Never unload the fallback and the failed revision in a way that leaves you cold on both.

The cost and procedure trade-off unique to serverless GPU

This is the decision the always-on playbook never had to make: during any canary you are paying for two warm GPU revisions at once. The trade-off is a warm-both window versus cold-start risk on rollback.

• Warm both revisions for the whole canary: instant rollback, but you pay for two accelerators until you promote or roll back.

• Let the fallback scale to zero: cheaper, but rollback now eats a 2-4 second cold start at the worst possible moment, mid-incident.

Bound the extra spend by keeping the warm-both window short and deliberate: warm the new revision, ramp quickly through your metric gates, and promote or roll back on a timer rather than leaving both warm indefinitely. GPU seconds are the unit to reason in; a basic CPU app on Cerebrium bills at "$0.000026 per second," and GPU is a multiple of that, so a two-hour warm-both window is a line item you can price in advance. The whole reason to run this on managed serverless rather than a hand-rolled queue is leverage: Cerebrium's own reporting is that the approach cuts "GPU costs nearly in half" while "reducing engineering overhead by more than 90%." For deeper background on the model, see our primers on the serverless platform and serverless GPU cold starts, and on picking AWS alternatives for AI workloads.

The adaptation, in one line: pre-warm before eligibility, split by percentage, keep the fallback warm, roll back by routing, and put a timer on the warm-both window so the safety net does not become the bill.