SLM Inference Cost Per Request on a Single GPU

Cost per request on one GPU is a ratio: the card's hourly rate divided by how many requests it completes per hour. You cannot change the first number much on a single card, so every lever that matters acts on the second. Three decide it for small language models: how much memory a request consumes, how much compute each generated token spends, and how efficiently your serving stack turns the GPU into finished requests. This page gives the short answer to each, and links down to the deeper measurements where we have them.

Lever 1: KV-cache footprint sets concurrency

File size tells you whether a model fits on the card once. KV-cache footprint tells you how many requests fit at the same time, and that is what actually divides your GPU-hour across requests. The two numbers come apart hard. In a public measurement of 8B models whose weights sit "within a gigabyte of each other," the KV cache at 32K context spanned "a factor of thirteen" across models. The architecture, not the download size, decides how many concurrent requests you can pack.

The reason is which layers grow a cache. The LFM2.5 8B-A1B measurement describes a model with "24 layers, of which 6 are full attention and 18 are short-convolution layers that keep a constant-size state... Only the 6 attention layers grow a KV cache with context, so 32K tokens cost 0.38 GB." A dense-attention model of similar size pays for every layer, every token. When concurrency is your cost denominator, the cache profile is the number to size against. The same memory discipline drives mixture-of-experts VRAM footprint on serverless GPU.

Lever 2: the vocabulary tax on every token

Even a small model pays a per-token cost most people never measure: scoring the output vocabulary at each generated token. On a Qwen3-4B essay grader, the VocabX write-up notes that "at every generated token the model scored a vocabulary of 151,669 entries, including many this task almost never uses." For a domain-specific workload, most of that compute is wasted.

Specializing the vocabulary to the tokens a task actually produces reclaims it. VocabX "cut measured inference cost by 77% without changing the base model" on that Qwen3-4B model. That is the largest single lever on this page, and it touches neither the GPU nor the weights - it narrows what the model has to compute per token. If you are tracking this yourself, our guide to token usage and cost per request covers the instrumentation.

Lever 3: the serving framework

The same model on the same GPU produces wildly different request counts per hour depending on the serving stack. For single-GPU Gemma 7B, Runpod reports that "vLLM significantly outperforms other frameworks, offering 24 times the throughput of Hugging Face Transformers and 3.5 times that of Hugging Face's Text Generation Inference (TGI)". Throughput is the denominator of cost per request, so a 24x gap is a 24x gap in GPU-hours spent per request. Tuning the server further is its own exercise - see vLLM server parameters for inference throughput.

Sizing the model to the card

Budget against usable memory, not the sticker. The LFM2.5 measurement is explicit that "our verdicts use the card's usable memory (an RTX 3070 has 7.0 GB usable of 8), not the sticker." Subtract weights and system overhead first; what remains is your KV-cache budget, and the cache profile from Lever 1 tells you how many requests that budget buys.

For production headroom on a dense 7B, Runpod "recommend[s] the 48GB GPU for the 7B model to account for model storage and system needs". That is the weights-plus-overhead floor before you add any concurrent-request cache at all.

Putting it together

In our testing the cost-per-request figure that matters is never the GPU's hourly rate in isolation - it is that rate divided by completed requests, and the denominator moves far more than the numerator. Pick an architecture whose KV cache stays small under your real context length, narrow the vocabulary to the task, and serve it on a framework that saturates the card. For the broader economics across model sizes and cluster shapes, see LLM inference cost at scale, and for quantization headroom on bigger cards, 4-bit LLM inference on H100.