Speculative Decoding on Serverless GPUs

Speculative decoding is one of the rare LLM optimizations that improves latency and per-token cost at the same time. But almost every number you have read was measured on a dedicated GPU running near-continuous batches. On a pay-per-use serverless GPU the accounting is different: you rent the draft model's VRAM and compute by the second, you pay for cold starts, and you rarely run the dense batches those benchmarks assume. This guide models what changes and gives you go/no-go conditions.

The short answer

Speculative decoding still wins on serverless GPU when your acceptance rate is high, your outputs are long, and your endpoint stays warm enough that cold starts are amortized over many tokens. It silently loses money when acceptance is low, outputs are short, or a mostly-idle endpoint pays a cold start per request. The draft model is not free: it occupies VRAM-seconds and adds forward passes you are billed for whether or not its tokens are accepted.

Why every published number is a dedicated-GPU number

The foundational result set the tone. The original paper reports that its authors "demonstrate it on T5-XXL and show a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs." Production write-ups since have followed the same experimental frame. One production guide on dedicated H100s benchmarks standard decoding against a draft model - "Standard decoding | ~1,200 | ~45 ms | ~$0.46 | High concurrency (32+ req), batch jobs. Draft model (Llama 3.2 1B) | ~2,600 | ~20 ms | ~$0.21 | Low-concurrency chat, interactive APIs" - and puts "EAGLE-3 | ~3,600 | ~15 ms | ~$0.15 | Instruction-following, coding, agents" at the top of the table. AWS likewise notes that "EAGLE achieves 2–3× speedups over standard autoregressive decoding and is widely deployed in production inference frameworks including vLLM", with parallel EAGLE going further: "P-EAGLE removes this ceiling by generating all K draft tokens in a single forward pass, delivering up to 1.69x speedup over vanilla EAGLE-3 on real workloads on NVIDIA B200."

Those are real gains - on a card you have already paid for, for a full hour, at high concurrency. None of them tell you what the draft model costs when you are billed by the GPU-second and serve one request at a time.

How serverless billing changes the math

On pay-per-use infrastructure you are charged for GPU-seconds, not tokens. That inverts several assumptions. The draft model occupies VRAM the whole time your endpoint is live and runs forward passes on every step; its overhead only pays off when its cheap tokens are accepted. Acceptance is everything. The same production guide explains that "The expected throughput gain depends on the acceptance rate alpha. If alpha is 0.8 (the draft model is right 80% of the time), expected accepted tokens per step is (1 - alpha^(N+1)) / (1 - alpha). At alpha = 0.8 with N = 5, you accept about 3.7 tokens per target model forward pass instead of 1."

Cold starts are billed and frequent on scale-to-zero endpoints. In our testing a TensorRT-LLM Llama 3 8B engine shows that "This code will run on every cold start and takes roughly ~10-15s to load the model into GPU memory." Speculative decoding means loading two sets of weights, not one. And serverless traffic is often batch-of-1 - exactly where speculative decoding helps most, but also where the dedicated-GPU throughput figures, gathered at large batch, do not apply. For how these charges accumulate, see our guide to LLM inference cost at scale and the economics of a serverless platform.

A cost-reasoning framework

Work the decision from acceptance rate outward. EAGLE-3's feature fusion lifts acceptance: "Acceptance rate on instruction-following tasks goes from 0.72-0.78 (Eagle-2) to 0.80-0.88 (Eagle-3) on Llama and Qwen families." The variables that decide whether the speedup survives the invoice: acceptance rate (higher means fewer billed target passes per token), draft/target latency ratio (the draft must be genuinely cheap), output length (fixed per-request costs amortize over long generations), cold-start frequency (a warm endpoint spreads the load cost over thousands of tokens), and per-second price (which turns every saved forward pass into a real number - convert throughput into a rate, the way we do for the B200's token economics).

Break-even is simple: the GPU-seconds saved by accepted tokens must exceed the draft VRAM-seconds, the rejected-draft compute, and the incremental cold-start weight load. If it does not at your traffic shape, the speedup is a latency win only, not a cost win. Serverless benchmarks do exist - one playbook reports that on Runpod you can "Add speculative decoding on a 3-GPU configuration and generation latency drops by another 45%, bringing TTFT to ~210ms at $0.39/M tokens." - but even that is a blended figure, not a draft-model-only cost line.

When it wins vs. silently costs money

Green light: interactive chat, coding assistants, or agents at batch-of-1; acceptance above ~0.8 on your actual prompts; long outputs and an endpoint kept warm. Red light: bursty, low-volume traffic that pays a cold start almost every request; short completions where fixed per-request costs dominate; low acceptance on out-of-distribution prompts, where you pay draft compute for rejected tokens. The trap is that latency improves on the dashboard while cost per token rises on the invoice.

Measuring it correctly on serverless

Do not trust vendor throughput specs; measure the full round trip on the platform you will bill against. Our own benchmarks were run under a stated methodology: "The following benchmark was done with 256 input tokens on 1xH100 of batch size 1. This was the total roundtrip time of a request made to the Cerebrium platform." On that basis we measured that "From our tests, it seems that vLLM is the best framework for use cases looking for the lowest TTFT of 123ms which is extremely quick if you take into consideration network latency." and, for throughput, that "As for throughput use cases, it seems SGLang was a clear winner achieving a throughput of 460 tokens per second on a batch size of 64." On a single A10 with TensorRT-LLM we found "In this tutorial we will achieve ~1700 output tokens per second (FP8)on a single Nvidia A10 instance however you can go up to ~4500 output tokens per second on a single Nvidia A100 40GB instance or even ~19,000 tokens on a H100." Reproduce that discipline: separate cold-start latency from warm steady-state, log acceptance rate per prompt class, and price both against your GPU-second rate. Network overhead matters too - see network latency in the inference pipeline.

Practical levers

Size the draft down and quantize it to cut VRAM-seconds and per-step cost. Keep a minimum replica warm to remove the per-request cold start that most often kills the cost case. Use n-gram or prompt-lookup drafting for repetitive outputs to avoid a second set of weights entirely. And route long, high-acceptance requests down the speculative path while sending short or low-acceptance ones down plain decoding, as you would when routing agentic GPU workloads. Providers vary widely in cold-start behavior and per-second pricing, so compare serverless providers on those axes specifically.

Key takeaways

Speculative decoding's 2-3x throughput win is real but was measured on dedicated GPUs at high batch. On pay-per-use serverless GPU the draft model is a metered cost. Model the break-even from acceptance rate, draft/target ratio, output length, cold-start frequency, and per-second price before you ship - then keep endpoints warm, size the draft down, and route by acceptance.