LLM JSON Mode: A Structured-Output Benchmark (2026)

Any production pipeline that feeds a language model’s output into another program needs that output to be valid, schema-conformant JSON every single time. This LLM JSON mode benchmark examines how the major approaches — prompt-only JSON, function and tool calling, and constrained decoding via grammars — actually behave under load, and how you would measure them honestly. The interesting questions are not “does it produce JSON” but “what fraction parses, how much latency does the guarantee cost, and what happens to throughput when you serve hundreds of concurrent requests against the same schema.” Those are engineering questions with measurable answers, and they rarely match marketing claims.

This post is methodology-first. The numbers we show are explicitly representative, drawn from published reports and our test-harness design, not freshly measured vendor results. The value is in the harness and the interpretation, both of which you can reproduce.

What this covers: the methods, the constrained-decoding internals, a benchmark harness you can run, representative results, and the trade-offs that bite in production.

Context

Getting JSON out of an LLM looks trivial until you put it behind an API with a service-level objective. A token sampler is a probability distribution over a vocabulary; nothing in the base decoding loop knows that the seventh character of a string field cannot be an unescaped quote, or that an enum admits only three values. The model has learned the shape of JSON from pretraining, so it usually emits something parseable. “Usually” is the problem. At one percent malformed output, a service handling a million requests a day breaks ten thousand times.

The failure modes are specific and recurring. Models truncate before the closing brace when they hit a token limit mid-object. They emit trailing commas, smart quotes, or markdown fences around the payload. They hallucinate keys that are not in the schema, or omit required ones. They produce a string where an integer was demanded, or a number formatted as 1,000. Under chain-of-thought prompting they prepend prose — “Here is the JSON you requested:” — that breaks naive parsers. Each of these is recoverable with enough retry logic, but retries cost latency and tokens, and they do not converge: a model that is confused about the schema stays confused.

Three structural pressures make this harder in 2026. Schemas are bigger and more nested, often generated from existing Pydantic or TypeScript types with deep unions and recursion. Tail latency matters more as structured output sits on the critical path of agentic loops, where one slow step compounds across many. And teams want to swap models freely, which means the validity guarantee cannot depend on a single vendor’s behavior. That combination is what pushed constrained decoding from a research curiosity into a default. For the surrounding validation layer — schema checks, repair, and guardrails that sit around whichever generation method you pick — see our companion guide on structured-output validation and guardrails.

Methods compared

There are four families of approaches, and they differ in where the constraint lives: in the prompt, in the API contract, in the decoder, or in a managed vendor layer.

Figure 1: A taxonomy of structured-output methods grouped by where the schema constraint is enforced — prompt, API contract, decoder, or vendor service.

The methods are not mutually exclusive. A real system might use vendor structured outputs for one model and self-hosted constrained decoding for another, behind one validation layer. But they have distinct cost and reliability profiles, so it pays to understand each on its own.

Prompt-only JSON

The baseline: describe the schema in the system prompt, optionally include few-shot examples, and ask for JSON. It works with any model, requires no special inference stack, and adds zero decode-time overhead. It is also the least reliable. Validity depends entirely on how well the model internalized the schema, which degrades with schema size, nesting depth, and competing instructions. Prompt-only is the right choice for quick prototypes, for models behind APIs that expose nothing better, and for cases where occasional malformed output is cheap to retry. It is the wrong choice for a tight SLO. In our representative figures it is also the most variable across models and across runs, which is the real liability — you cannot reason about a system whose validity rate swings by model and prompt phrasing.

Function and tool calling

Tool calling reframes structured output as a function signature. You declare a tool with a JSON Schema for its arguments; the model “calls” it; the runtime returns the arguments as JSON. Most hosted models were fine-tuned heavily on this format, so validity is markedly better than free-form prompting, and it carries clean semantics — the JSON is unambiguously the answer, not buried in prose. The catch is that conformance is still learned, not enforced. The model can still omit a required field or violate a constraint the fine-tuning never emphasized, especially on deep nesting or unusual types. Tool calling is an excellent default for hosted models and is what most agent frameworks lean on. Just do not mistake “high validity” for “guaranteed validity” — you still need a validator behind it.

Constrained decoding

Constrained decoding enforces the schema at the only place that can offer a guarantee: the token sampler. The schema is compiled into a state machine, and at every decode step the decoder computes which next tokens keep the output on a path that can still complete as valid JSON. Tokens that would break the grammar are masked to probability zero before sampling. Done right, the schema-validity rate is effectively 100 percent — invalid output is not merely unlikely, it is unreachable.

The libraries differ in how they build and apply that mask. Outlines compiles a regex or JSON Schema into a finite-state machine and precomputes token-to-state transitions. XGrammar uses a context-free grammar with a pushdown automaton and aggressive precomputation of a context-independent token cache, targeting near-zero per-token overhead. llguidance takes a similar grammar-based route with a fast lexer and is wired into several serving stacks. All three integrate with engines like vLLM, which exposes a guided_json interface so you can pass a schema directly to the server.

Figure 2: Constrained decoding compiles a schema into an automaton, derives a per-token allowed mask, applies it to the model logits, then advances the automaton state after each sampled token.

Two costs come with the guarantee. First, grammar compilation: turning a schema into an automaton plus token caches takes time, from milliseconds for a flat schema to hundreds of milliseconds or more for deeply recursive ones. Pay it once and cache it, and it amortizes; pay it on every cold request and your time-to-first-token spikes. Second, the token-vocabulary alignment problem: JSON grammars are defined over characters, but models emit subword tokens, so the mask must be computed over which tokens are legal given the remaining grammar — the engineering that the libraries above spend most of their effort on. When that mapping is precomputed well, per-token overhead is small; when it is not, throughput suffers.

Vendor structured modes

OpenAI’s Structured Outputs, Google’s Gemini response-schema mode, and similar offerings are essentially managed constrained decoding. You supply a JSON Schema; the provider guarantees conformant output. You get the validity guarantee with none of the operational burden, at the cost of supported-schema restrictions (recursion limits, unsupported keywords) and zero visibility into the latency the constraint adds. For teams already on a hosted model, this is often the pragmatic default. For self-hosted or multi-model fleets, you are back to running Outlines, XGrammar, or llguidance yourself.

Benchmark methodology

A credible LLM JSON mode benchmark has to separate four things that get conflated: can it produce valid JSON, is the content correct, how fast, and how much does the constraint cost. Each maps to a distinct metric, and a method can win on one while losing on another.

Figure 3: The benchmark harness routes a schema-annotated dataset through pluggable method adapters into a shared inference backend, then scores every generation for schema validity and semantic accuracy while a separate probe records latency and throughput.

Metrics. Five carry the analysis:

• Schema-validity rate — fraction of outputs that parse and validate against the target JSON Schema. The headline reliability number.
• Semantic accuracy — of the valid outputs, the fraction whose content is correct (right values, right extraction). Validity without accuracy is worthless; a method can emit perfectly-shaped wrong answers.
• Throughput (tokens/sec) — sustained generation rate under realistic concurrency, the cost axis that matters at scale.
• Time-to-first-token (TTFT) — latency before the first byte, where grammar-compilation cost shows up, especially on cold schemas.
• Latency overhead — end-to-end constrained latency minus the unconstrained baseline on identical inputs, isolating the price of the guarantee.

Harness design. Keep one inference backend fixed and swap only the method adapter, so differences are attributable to the method rather than the engine. Each adapter takes a prompt plus schema and returns raw text. A shared validator runs the same JSON Schema check across all methods — never trust a method’s self-report of validity. Semantic scoring is task-specific: exact match for extraction, field-level F1 for partial credit, or an LLM-judge for open-ended fields. A separate probe records TTFT and throughput so scoring overhead never contaminates timing. Crucially, measure cold versus warm grammar compilation separately — first-request and steady-state are different regimes and reporting only one is misleading.

Datasets. Mix three workloads: flat schemas (5–10 scalar fields, the common extraction case), nested schemas (objects, arrays, unions, the realistic case), and deeply recursive schemas (trees, ASTs, the stress case where compilation cost and validity gaps both surface). Pair each with a ground-truth task so semantic accuracy is meaningful, not just structural.

Hardware and controls. Pin the GPU, driver, engine version, and library versions; report them. Fix temperature, max tokens, and concurrency. Run enough requests per cell that validity-rate confidence intervals are tight, and report variance, not just means — tail behavior is the point. Warm up before measuring steady-state throughput. For the surrounding engine-level methodology — how to benchmark the serving layer itself across vLLM, TGI, and SGLang — see our LLM inference benchmark.

Results and interpretation

The table below is representative and illustrative, synthesized from published reports and the harness design above. It is not a set of freshly measured vendor results, and you should not cite the cells as measured numbers. The point is the direction and shape of the differences, which are stable across sources; the exact values depend on model, schema, hardware, and library version.

Figure 4: An interpretation map of how each metric behaves across methods — constrained decoding dominates validity, semantic accuracy is governed by schema design rather than method, and grammar caching is what protects time-to-first-token.

Representative / illustrative comparison — directional only, not measured-as-stated:

Method	Schema-validity rate	Semantic accuracy (of valid)	Throughput vs. baseline	Cold TTFT overhead	Warm TTFT overhead
Prompt-only JSON	~85–95%, high variance	Baseline	~1.0x (none)	None	None
Function / tool calling	~95–99%	≈ baseline	~0.97–1.0x	Minimal	Minimal
Constrained decoding (cached grammar)	~99.9–100%	≈ baseline	~0.9–1.0x	High (compile)	Low
Vendor structured mode	~99.9–100%	≈ baseline	Opaque	Opaque	Opaque

Read the table by column, not by row. Validity is where constrained decoding and vendor modes separate decisively from the rest: they make malformed output unreachable, while prompt-only and tool calling leave a residual failure rate that, however small, must be handled downstream. Semantic accuracy is roughly method-neutral — constraining the shape of output does not make the content more correct, and in pathological cases an over-tight grammar can even nudge a model toward a valid-but-wrong answer. Accuracy is dominated by schema design and prompt quality, not by the enforcement mechanism. Throughput for well-implemented cached constrained decoding sits close to baseline; the modern libraries have driven per-token masking overhead low enough that it is no longer the headline cost. TTFT is the real watch-item: cold grammar compilation can dominate first-request latency, which is precisely why caching compiled grammars per schema is the single most important production optimization. Warm steady-state TTFT overhead is small.

The one-line takeaway: if you need guaranteed valid JSON, constrained decoding or a vendor structured mode gets you there at modest steady-state cost, but you must engineer around cold-start compilation — and you still need a separate accuracy strategy, because no enforcement method makes wrong answers right.

Trade-offs and what goes wrong

The first failure is cold-start compilation latency. A service that compiles a fresh grammar on every request, or evicts its grammar cache under memory pressure, will show pathological p99 TTFT even though its median looks fine. Cache compiled grammars keyed by schema, size the cache for your schema cardinality, and pre-warm the schemas you know you will use.

The second is the validity-versus-accuracy trap. Teams adopt constrained decoding, watch the validity rate hit 100 percent, and declare victory — then discover extraction accuracy did not move, because the model was already producing well-shaped output and its errors were semantic. Worse, a constraint can mask a confused model: forced down a valid path, it confabulates a plausible value rather than signaling uncertainty. Always measure semantic accuracy separately, and leave the model an escape hatch (a nullable field, an explicit “unknown” enum) so it is not forced to invent.

Third, schema complexity hits walls. Deeply recursive or very large schemas inflate compilation time and can exceed vendor structured-mode limits on recursion or unsupported keywords. Flatten schemas where you can, bound recursion depth, and validate that your schema compiles before it reaches production.

Fourth, tokenizer and grammar edge cases. The token-to-character alignment that constrained decoding depends on has corner cases — Unicode, whitespace handling, number formats — that can let a technically-invalid string slip through or, more often, reject a valid one. Pin library versions and keep a downstream validator regardless; constrained decoding reduces the failure rate to near zero, but “near zero” is not “audited.”

Fifth, interaction with caching and batching. Per-request grammars complicate KV-cache reuse and continuous batching, since each sequence carries its own automaton state. The cost is usually small with modern engines, but it is real, and it interacts with prefix-cache strategy — see our notes on KV-cache optimization for LLM inference.

Practical recommendations

Match the method to the constraint you actually have.

• Prototyping, or behind an API with nothing better: prompt-only JSON, with a validator and bounded retries. Cheap, portable, good enough for low-stakes paths.
• Hosted model, moderate reliability needs: tool/function calling with a strict downstream validator. High validity, clean semantics, near-zero overhead, no infrastructure to run.
• Self-hosted, strict SLO: constrained decoding via vLLM with Outlines, XGrammar, or llguidance. Cache compiled grammars, pre-warm hot schemas, and separate cold from warm TTFT in monitoring.
• Hosted model, strict SLO, simple schemas: vendor structured outputs. The guarantee for free, accepting schema restrictions and opaque latency.

Three rules apply across all of them. Always run an independent schema validator behind whatever generation method you choose — defense in depth, since even constrained decoding has edge cases. Always measure semantic accuracy as a first-class metric, never inferred from validity. And always benchmark on your schemas and hardware, because the representative numbers here tell you the shape of the trade-off, not the value you will see.

FAQ

Does JSON mode guarantee valid JSON?
Not by itself. Prompt-only “JSON mode” and tool calling make valid output highly likely but not guaranteed — a residual fraction is malformed or schema-violating. Only constrained decoding and managed vendor structured outputs, which enforce the schema at the token sampler, drive validity to effectively 100 percent. Even then, keep a downstream validator for tokenizer and edge-case failures. Treat “JSON mode” as risk reduction unless it is grammar-enforced.

Does constrained decoding slow down inference?
Steady-state, only modestly. Modern libraries like XGrammar and llguidance precompute token masks so per-token overhead is small, and cached-grammar throughput sits close to the unconstrained baseline. The real cost is cold grammar compilation, which inflates time-to-first-token on the first request for a new schema. Cache compiled grammars and pre-warm hot schemas, and the overhead becomes negligible in production.

Constrained decoding or function calling — which should I use?
Function calling is the pragmatic default for hosted models: high validity, clean semantics, no infrastructure. Constrained decoding is the choice when you self-host and need a hard guarantee, since it makes invalid output unreachable rather than merely unlikely. If you are on a hosted API with a structured-output mode, that mode gives you the constrained-decoding guarantee without operating the stack yourself.

Why does my model produce valid JSON but wrong answers?
Because validity and accuracy are independent. Constraining the shape of output does nothing for the content — a model that misreads the input emits a well-formed wrong answer. Constraints can even worsen this by forcing a confused model down a valid path instead of letting it signal uncertainty. Fix it with better prompts and schema design, and by giving the model a nullable or “unknown” option.

How do I benchmark structured output fairly?
Hold the inference backend fixed and swap only the method adapter, so differences are attributable to the method. Validate every output with one independent schema checker rather than trusting self-reports. Measure validity, semantic accuracy, throughput, and TTFT separately, and report cold versus warm grammar compilation distinctly. Use flat, nested, and recursive schemas, pin all versions and hardware, and report variance, not just means.

Further reading

• LLM output validation and structured-output guardrails (2026) — the validation and repair layer that wraps any generation method.
• LLM inference benchmark: vLLM, TGI, SGLang, Triton (Q2 2026) — engine-level methodology for the serving layer underneath.
• KV-cache optimization for LLM inference (2026) — how per-request grammars interact with cache reuse and batching.
• Outlines and XGrammar — open-source constrained-decoding libraries.
• vLLM guided decoding documentation — the guided_json serving interface used in the harness.