Tool Call Schema Accuracy with Constrained Decoding in vLLM

The problem: tool calls that pass most of the time

If you run agents on a self-hosted vLLM endpoint, you have probably seen tool calls fail validation for no obvious reason. The same prompt that works 24 times produces a malformed arguments object on the 25th. The model is right about which tool to call and roughly what to pass, but the serialized output is not valid against the schema your agent framework expects.

This is not rare. When Ubicloud served Kimi K2.6 with vLLM 0.28, they got the expected tool-call schema only 92% of the time - "2 out of every 25 attempted tool calls failed." For a single-step chatbot a 92% rate is annoying. For a multi-step agent it is corrosive: a five-call plan has roughly a one-in-three chance that at least one call is malformed, and a malformed call either crashes the loop or forces an expensive retry.

The root cause is the default generation path. The engine samples tokens freely and validates the result after the fact. Nothing during generation prevents the model from emitting a stray token that breaks the JSON.

What constrained decoding actually does

Constrained decoding closes that gap. Instead of sampling freely and checking later, the inference engine guides generation step by step: at each step, only valid tokens are allowed. Tokens that would produce a syntactically invalid tool call are masked out before sampling ever happens.

In practice the generation is forced through an explicit sequence of phases - "Begin tool call → Tool name → Tool call index → Begin arguments → Schema-constrained JSON → End tool call" - and invalid paths are blocked during generation. The model still chooses the next token; it just chooses from a restricted, syntactically-correct set.

The payoff is a change in kind, not degree. As Ubicloud put it, this "changes schema accuracy from a best-effort approach to a runtime guarantee." Ubicloud re-ran the K2VV ToolCall benchmark with constrained decoding enabled and verified 100% tool-call schema accuracy across all served model versions for Kimi and DeepSeek before shipping it to production. The overhead is negligible relative to the retries it eliminates.

The honest caveat: valid is not the same as correct

Constrained decoding guarantees syntax, not semantics. It ensures the JSON parses and matches the schema. It does not ensure the model picked the right tool, passed the right values, or reasoned correctly about the task. A perfectly-formed call to the wrong function is still wrong.

So treat constrained decoding as removing one entire class of failure - malformed output - not as a quality guarantee. You still need evals for tool selection and argument correctness. What you gain is that those evals stop being contaminated by parse errors, so a failed test now means a reasoning problem you can actually debug. This is the same discipline we apply to serving fine-tuned LLMs: separate the plumbing from the model quality.

Enabling it in vLLM

vLLM exposes tool parsing through server flags. You enable automatic tool choice and point the engine at the parser that matches your model family. From vLLM's own Flint documentation, a working invocation looks like:

vllm serve RecursiveSI/flint \
  --enable-auto-tool-choice \
  --tool-call-parser glm47 \
  --reasoning-parser glm47 \
  --exclude-tools-when-tool-choice-none

The --tool-call-parser flag is model-specific - glm47 for that model, a different value for Kimi, DeepSeek, or Llama. The parser is what lets vLLM understand the model's tool-call grammar and apply schema constraints during decoding. If you are tuning the rest of the serving configuration too, our rundown of vLLM server parameters for throughput covers the flags that interact with batching and latency.

Because constrained decoding only matters once you expose tools, it pairs naturally with an OpenAI-compatible endpoint for open-source LLMs, which is how most agent frameworks expect to issue tool calls. If you are standing up DeepSeek specifically, our serverless DeepSeek-R1 guide walks the same OpenAI-compatible path end to end.

Verifying the fix with a baseline test

Do not take 100% on faith - measure it on your own deployment. Baseline-testing tools for self-hosted LLM setups check exactly the behavior that matters here: as the Giraffe tool describes, they verify "structured JSON and tool-call behavior," response times and streaming pauses under load, and how short requests behave when long prompts arrive. The method is simple: record a baseline before enabling constrained decoding, flip the flag, and re-run against the same saved baseline. You want to see schema validity climb to 100% while latency stays flat.

Latency flat is the part people worry about. In our testing of vLLM, SGLang and TensorRT for a Llama 3.1 API, vLLM delivered the lowest TTFT at 123ms on a single H100 - there is plenty of headroom to absorb the small cost of token masking without breaking an agent's responsiveness budget.

Rollout checklist

• Measure the baseline. Run your real tool schemas through the unconstrained endpoint and record the schema-validity rate. If it is anywhere near 92%, you have a problem worth fixing.

• Pick the right parser. Match --tool-call-parser to your model family; the wrong parser silently degrades results.

• Enable constrained decoding and re-benchmark. Confirm schema validity hits 100% and that TTFT and throughput are unchanged.

• Keep a semantic eval. Validity is table stakes; separately track tool-selection and argument correctness.

• Canary before full rollout. Ship the config change behind a canary rollout with fast rollback and promote it through staged model versions once the numbers hold.

Agent workloads are bursty and latency-sensitive, which is why teams run them on autoscaling GPU infrastructure - DistilLabs, for example, runs up to 150 requests per second per model at peak on Cerebrium. If you are sizing hardware for this pattern, our guide to GPU inference for AI agent workloads covers the tradeoffs. Constrained decoding is the cheapest reliability win in the stack: one flag turns a flaky 92% into a dependable 100%.