MiniMax-M3.1-Flash: GPU Memory & TTFT Runbook

TL;DR

MiniMax-M3.1-Flash is a mixture-of-experts model that ships as an approximately 252 GB Hugging Face checkpoint spread across 72 files, so your first deployment is fundamentally a multi-GPU exercise. The reference configurations run it on 4x NVIDIA B200 or 4x NVIDIA GB300 at tensor parallelism 4 (TP4) with an expert-parallel worker layout, and a single node can serve up to 256 concurrent requests against its long context window.

On time to first token (TTFT): the single biggest lever in the serving stack is QKV GEMM + norm + RoPE + KV-insert fusion, which delivers about 19% lower P50 TTFT at TP4 and 1.1–1.6x faster prefill at 512+ tokens. The rest of this runbook explains the memory math, the supported hardware, what actually drives TTFT, and the failures that bite on day one.

What MiniMax-M3.1-Flash is, in deployment terms

Forget the model card for a moment and look at it as an operator. MiniMax-M3.1-Flash is a large MoE model whose weights alone dominate your VRAM budget - the ~252 GB checkpoint does not fit on any single accelerator available today, so sharding is not optional. That makes it closer to a Llama-405B-class deployment than a single-card model. If you have never sharded a model this size before, our walkthrough of Llama 405B serverless GPU inference covers the same multi-GPU reasoning, and the MoE VRAM footprint guide explains why expert layers change the memory picture versus a dense model.

GPU memory requirements

Three things consume GPU memory on this model, in order of magnitude:

1. Weights. The ~252 GB checkpoint (72 files) is the floor. At TP4 that is roughly 63 GB of weights per GPU before any runtime overhead, which is why the reference hardware is 180 GB+ class accelerators.

2. KV cache. With a context this long, KV cache is the variable that decides how many concurrent requests you can actually hold. The recipes use FP8 KV cache on vLLM and SGLang to shrink that footprint, and NVFP4 KV cache is available as a more aggressive option.

3. Activations and fragmentation. Leave headroom; a checkpoint that technically fits will still OOM under concurrency if you size KV cache to the edge.

When you are planning the download and load step, remember that a 252 GB checkpoint is also a cold-start problem, not just a memory problem - see loading model weights from object storage for how pull time scales with checkpoint size.

Supported GPUs and parallelism

The published reference configurations target 4x B200 or 4x GB300 per node, TP4, with an expert-parallel worker setup for the MoE layers. Throughput scales hard with the hardware generation: agentic serving reaches up to 418.9 tok/s per user on GB300 at the latency-sensitive end, and up to 724.9 tok/s per user with 191.5K total tok/s per GPU on NVIDIA Rubin (SM107).

If you are still choosing silicon, our NVIDIA B200 throughput and TTFT page and the broader 2026 GPU buyer's guide put these numbers in context against H100/H200-class cards. For the networking side of a four-GPU node and beyond, see multi-node LLM inference.

Time to first token: what drives it

TTFT on a sharded MoE is dominated by the prefill pass and the attention projection kernels. The optimizations that matter most here are:

• QKV GEMM + norm + RoPE + KV-insert fusion - about 19% lower P50 TTFT at TP4 and 1.1–1.6x faster prefill at 512+ tokens.

• NVFP4 MegaMoE with MXFP8 shared-expert fusion - 20% lower kernel latency in prefill and 7% higher prefill throughput, via early TMEM release.

The headline here is that TTFT is a software number as much as a hardware one: the same GPU gives you materially different first-token latency depending on whether these fusions are compiled in. When you benchmark your own deployment, measure the full roundtrip, not just the kernel. In our testing of vLLM, SGLang and TensorRT-LLM on Llama 3.1, we measured TTFT as the total roundtrip of a request to the platform (256 input tokens, 1xH100, batch size 1) - vendor kernel numbers and end-to-end numbers are not the same thing, and only the latter predicts your p99. For splitting prefill and decode onto different workers, see disaggregated prefill/decode serving.

First-deployment runbook

1. Provision the node. 4x B200 or 4x GB300, TP4, expert-parallel workers.

2. Stage the checkpoint. Pull all 72 files (~252 GB) to fast local storage before first boot; this is your dominant cold-start cost. For comparison, in our TensorRT-LLM work a small Llama 3 8B engine took roughly 10–15s to load into GPU memory on each cold start - a 252 GB MoE is a different order of magnitude, so plan to keep replicas warm.

3. Pick a backend. vLLM, SGLang or TensorRT-LLM; enable FP8 KV cache on vLLM/SGLang from the start.

4. Enable the fusions. Confirm QKV GEMM fusion and NVFP4 MegaMoE/MXFP8 paths are active - this is where your 19% TTFT and 7% prefill throughput live. Tune the knobs with our vLLM server parameters guide.

5. Set concurrency. Start well below the 256-request ceiling and raise it while watching KV-cache pressure.

6. Define SLOs. Decide your TTFT and throughput targets up front - our goodput and latency SLO framework explains why goodput, not raw throughput, is the number to hold.

Tuning: latency vs throughput vs context

These three pull against each other. The GB300 figure of up to 418.9 tok/s per user is the latency-sensitive operating point; pushing batch size up trades per-user latency for aggregate throughput. KV-cache precision is the other axis - FP8 or NVFP4 KV cache buys you concurrency and context at the edge of numerical headroom. For multi-turn agents, prefix caching reclaims most of the prefill cost on repeated system prompts, which directly helps TTFT.

Common first-deployment failures

• OOM under concurrency. The weights fit but KV cache at 256 requests does not. Drop to FP8/NVFP4 KV cache or lower max concurrency.

• TTFT far above spec. The fusions are not compiled in, or you are measuring cold replicas. Warm the pool and confirm the kernel paths.

• Checkpoint pull dominates cold start. A 252 GB download on the critical path. Cache it locally and avoid scale-to-zero for this model class.

• Benchmarking the kernel, not the roundtrip. As noted above, our benchmarks use end-to-end roundtrip; kernel-only TTFT will flatter you and mislead capacity planning.

Autoscaling a model this size reliably is the hard part of production. DistilLabs runs up to 150 requests per second per model at peak on Cerebrium with 50% lower inference costs - the same autoscaling behaviour applies to large MoE endpoints.

Next steps

Decide your hardware (B200 vs GB300), fix your KV-cache precision, enable the fusion paths, and benchmark end-to-end before you commit a concurrency ceiling. If you are serving an OpenAI-compatible endpoint, our DeepSeek-R1 serverless guide shows the same pattern end to end.