Mistral Large 4: GPU Memory & TTFT Deploy Guide

TL;DR

Mistral Large 4 is a mixture-of-experts model with 1T total, 49B active parameters per token and a 1M-token context window. Two numbers decide your first deployment:

• GPU memory is governed by total parameters, not active. You must hold all 1T parameters in VRAM, so your choice of precision variant (FP8 vs NVFP4) and your GPU count are the levers that matter.

• Time to first token (TTFT) is governed by prefill. vLLM's prefill/decode context parallelism (PCP+DCP) cuts TTFT by up to 1.6x, and DSpark speculative decoding roughly doubles output throughput on TP8.

Mistral Large 4 has day-0 support in vLLM on both NVIDIA and AMD Instinct MI350 GPUs, so you can serve the open weights yourself rather than pay the API.

What a first deployment must actually solve

Two questions dominate the first week: will the model fit, and how fast does the first token come back. Everything else - autoscaling, cost, routing - follows once those are answered. The trap for a model this size is reasoning from the active-parameter count. 49B active sounds like it should fit on one or two cards; the 1T total says otherwise. Below we walk the two memory levers and the two latency levers, then the serving stack, accelerator choice, and the build-vs-buy math.

Weight loading is also part of first-token-on-a-cold-replica reality. For a far smaller model, Llama 3 8B on TensorRT-LLM, we measured roughly 10-15s to load the model into GPU memory on every cold start. A trillion-parameter model loads far more, so plan replica warm-up deliberately - we cover the mechanics in our guide to loading model weights from object storage and vLLM replica startup.

Memory lever #1 - MoE total vs active parameters

Mistral Large 4 routes each token to a small subset of experts, so only 49B of its 1T parameters are active per token. That lowers compute per token, not memory. Because any token can be routed to any expert, the serving engine must keep every expert resident in VRAM - it cannot lazily page experts in without wrecking latency. So your memory budget is sized by the full 1T weights, not the 49B.

We go deep on this mechanism, including why MoEs "require high VRAM as all experts are loaded in memory," in our mixture-of-experts VRAM footprint guide. The practical takeaway for deployment sizing: this is a multi-GPU, likely multi-node model regardless of precision. Our multi-node LLM inference guide covers the cluster shape.

Memory lever #2 - precision variant FP8 vs NVFP4

Mistral ships two official variants: FP8 and NVFP4. Precision is your single biggest memory lever on a fixed GPU count. NVFP4 (4-bit) roughly halves the weight footprint versus FP8 (8-bit), letting you fit the model on fewer cards or leave more room for KV cache at long context - which matters a lot at a 1M-token window.

The trade-off is the usual one: lower precision can shift accuracy and, depending on kernel support, throughput. Start with FP8 if your accelerator's FP8 path is the most mature and you have the cards; reach for NVFP4 when VRAM is the binding constraint. For background on sub-8-bit serving, see our 4-bit LLM inference on H100 guide.

TTFT lever #1 - PCP+DCP prefill (up to 1.6x)

TTFT is dominated by prefill: the model has to read your prompt before it emits token one, and at long contexts prefill is the long pole. vLLM's prefill context parallelism and decode context parallelism (PCP+DCP) cut TTFT by up to 1.6x for Mistral Large 4. If your workload sends long prompts - RAG, agents, document analysis - this is the lever with the largest effect on perceived latency.

The complementary architectural move is splitting prefill and decode onto different resources so a long prompt doesn't block ongoing generations. We explain that pattern in disaggregated prefill/decode LLM serving, and the SLO framing in LLM serving goodput and latency SLOs.

TTFT/throughput lever #2 - speculative decoding (~2x on TP8)

After the first token, decode throughput drives the rest of the stream. vLLM's DSpark speculative decoding gives about 2x output throughput on tensor parallelism size 8 (TP8). Speculative decoding drafts several tokens cheaply and verifies them in one pass, so it mostly helps tokens-per-second rather than TTFT directly - but higher throughput shortens queueing, which indirectly protects TTFT under load. Our speculative decoding throughput guide covers when the 2x holds and when it doesn't.

Serving stack: vLLM day-0

Mistral Large 4 received day-0 support in vLLM, which is the fastest path to a self-hosted endpoint: the PCP+DCP prefill and DSpark levers above are built in. Tune the server parameters - tensor/context parallel sizes, max model length, KV cache fraction - with our vLLM server parameters guide. To expose it cleanly to application code, stand up an OpenAI-compatible endpoint.

Accelerator choice: NVIDIA and AMD MI350 (ROCm)

vLLM runs Mistral Large 4 day-0 on AMD Instinct MI350 Series GPUs with ROCm, including DSpark, image inputs, and expert parallelism with MXFP4 experts. That means your first deployment is not locked to NVIDIA. AMD's large-VRAM parts are attractive precisely because this model is memory-bound; compare the economics in our MI300X vs H200 throughput analysis and the broader 2026 GPU Buyer's Guide.

Build vs buy: API price vs open weights

The case for self-hosting got stronger on October 21, when Mistral Large 4 API pricing was observed to double - input $0.68 to $1.36 and output $2.09 to $4.18 per 1M tokens. Because the model is released as open-weight under the Mistral Ecosystem Open-Weight License (an MIT derivative with limits for model-as-a-service offerings), you can run it yourself. Note that license caveat if you intend to resell inference.

Self-hosting a 1T MoE is not free either - it's a multi-GPU footprint - so the decision turns on volume. One Cerebrium customer, Distil Labs, runs up to 150 requests per second per model and cut inference costs 50% while raising accuracy from 83% to 92%; the full story is in our Distil Labs case study. Model the crossover with our LLM inference cost at scale guide.

First-deployment checklist

1. Size for 1T, not 49B. All experts stay resident; plan multi-GPU/multi-node VRAM.

2. Pick a precision variant. FP8 for mature kernels and headroom; NVFP4 when VRAM is the constraint.

3. Enable PCP+DCP in vLLM to cut TTFT by up to 1.6x on long prompts.

4. Turn on DSpark speculative decoding for ~2x output throughput on TP8.

5. Choose your accelerator - NVIDIA or AMD MI350/ROCm, both day-0.

6. Plan cold starts / weight loading before you enable scale-to-zero.

7. Run the build-vs-buy math against the doubled API price at your volume.