LLM Structured Output in 2026: JSON Mode, Constrained Decoding and Schema Adherence

Last Updated: October 2026

Most production failures in LLM-backed software are not hallucinations in the headline sense. They are a missing closing brace, a string where an integer was required, or a polite “Here is the JSON you asked for:” glued in front of the payload. A single malformed response can stall an agent loop, poison a queue, or silently drop a record. LLM structured output is the family of techniques that stops this, and in 2026 the techniques differ more than the marketing suggests.

This matters now because every major provider ships a schema-enforcing mode, every serious open-source inference engine ships a grammar backend, and teams are routing between both. “JSON mode” no longer means one thing. Choosing the wrong layer costs you either reliability, latency, or answer quality, and the trade-offs are poorly documented in one place.

You will leave with a clear mental model of where a schema constraint can live, how token-level masking actually works, what the official provider documentation says about limits, how to build a measurement harness that does not lie to you, and a validation-and-retry pattern you can run today.

What this covers: the four enforcement layers, constrained-decoding internals, verified provider behaviour, a benchmark harness with runnable Python, the quality cost of format restrictions, failure modes, and a decision checklist.

Context and Background

A language model produces text by repeatedly sampling one token from a probability distribution over its vocabulary. Nothing in that loop knows that a JSON string cannot contain a raw newline, or that a field typed as an enum admits exactly three values. The model has learned the shape of JSON from pretraining and from instruction tuning, so it usually emits something parseable. “Usually” is the entire problem: at a one percent malformed rate, a service handling a million calls a day breaks ten thousand times.

The industry went through three phases. First came prompt-only JSON, where you describe the schema in the system prompt and hope. Then came function and tool calling, where models were fine-tuned on a call-signature format and validity improved sharply, but nothing was guaranteed. Then came schema-enforced generation, where the decoder itself refuses to emit a token that would make the output unparseable or non-conformant. OpenAI’s documentation describes Structured Outputs as “the evolution of JSON mode” and recommends it whenever possible; the older json_object mode guarantees valid JSON but, per the same documentation, not adherence to a schema.

Two research threads shaped the current landscape. XGrammar (arXiv:2411.15100) showed that most of the grammar work can be moved ahead of time by splitting the vocabulary into tokens whose validity is context-independent and tokens that need a runtime check, and overlapping the remaining work with GPU execution. The authors report up to a 100x speedup over earlier structured-generation solutions and near-zero overhead end to end. Separately, JSONSchemaBench (arXiv:2501.10868) assembled roughly 10,000 real-world JSON schemas to compare six frameworks (Guidance, Outlines, llama.cpp, XGrammar, OpenAI and Gemini) on efficiency, constraint coverage and output quality. Its central lesson is that “supports JSON Schema” is a spectrum, not a boolean.

If you want the decoder internals in more depth, our companion piece on constrained decoding with XGrammar, Outlines and llguidance goes through grammar compilation and engine comparisons. This article is the wider decision guide: which layer to use, what it costs, and how to verify it on your own data. For the primary sources, the OpenAI Structured Outputs guide and the XGrammar technical report are the two I would read first.

One framing note. Valid JSON, schema-valid JSON and correct JSON are three different properties. A decoder can guarantee the first two. Only your validators, evaluators and tests can speak to the third. Most of the confusion in vendor comparisons comes from treating them as one.

Where the Constraint Lives: A Reference Architecture for LLM Structured Output

There are four places a schema can be enforced: in the prompt, in the model’s fine-tuned behaviour (tool calling), in a hosted provider’s decoder, or in a decoder you run yourself. Only the last two offer a guarantee, because only they intervene in token sampling. The first two are probabilistic and always need a validator and retry path behind them.

Figure 1: The four enforcement layers for LLM structured output. Prompt and tool calling are learned behaviour and converge on validate-and-retry; provider strict mode and self-hosted grammar engines mask tokens during decoding, leaving only semantic validation.

The diagram is a decision aid, not a ranking. Every path ends in a validator because even a perfect decoder cannot know that end_date must be after start_date. What changes by layer is how often the validator fires on syntax problems versus meaning problems, and how much retry budget you need to provision.

Layer one: prompt-only JSON

You describe the schema in text, perhaps with a few examples, and parse what comes back. It works on any model, adds no decoding overhead, and needs no special serving stack. It is also the layer whose reliability depends most on schema size, nesting depth, competing instructions and the specific model. The honest characterisation is that it is fine for prototypes and low-stakes enrichment, and a liability inside a service-level objective. Its worst property is not its average failure rate but its variance: you cannot reason about a system whose validity swings with a prompt edit or a model upgrade.

Layer two: tool and function calling

Tool calling recasts the output as the arguments of a function. You declare a JSON Schema for the parameters, the model “calls” the function, and your runtime receives arguments as JSON. Because models were heavily tuned on this format, validity is far better than free-form prompting, and the semantics are clean: the JSON is the answer, not text decorated with prose.

Conformance is still learned rather than enforced in the plain mode. Strict variants change that. Anthropic’s documentation describes a strict: true flag on tool definitions that validates tool names and inputs against the schema, and it can be combined with JSON outputs in the same request. A subtle scope detail from that documentation: the grammar applies to the model’s direct output, not to tool results or thinking blocks. Use tool calling when the model must trigger an action; use a response format when you want the reply itself to take a shape.

Layer three: provider-hosted schema enforcement

Here the provider compiles your schema into a grammar and applies it inside their serving stack. OpenAI exposes this as json_schema with strict: true in response_format (Chat Completions) or text.format (Responses API). Anthropic exposes output_config.format with type: "json_schema"; its current documentation says beta headers are no longer required, and the older output_format parameter works only with the structured-outputs-2025-11-13 beta header. Google’s Gemini API accepts a JSON schema with a JSON MIME type through its structured-output interface and supports Pydantic and Zod definitions in its SDKs.

You give up control. You cannot see the mask, you cannot tune the grammar, and you inherit the provider’s supported-schema subset, which the next sections detail. In exchange you get a guarantee without running any inference infrastructure, and the provider absorbs grammar-engine upgrades.

Layer four: self-hosted grammar engines

If you serve open-weight models, the schema constraint is a flag on your inference server. XGrammar’s repository describes itself as the default structured-generation backend for most LLM inference engines and names vLLM, SGLang, TensorRT-LLM and MLC-LLM among its integrations. llguidance, a Rust library with Python and C interfaces, lists integrations including SGLang, vLLM, llama.cpp via a build flag, mistral.rs and onnxruntime-genai, and its README lists OpenAI’s Structured Outputs (JSON Schema only) among its integrations. Outlines is the other widely used option, with a regex and finite-state-machine heritage.

Self-hosting gives you the most control: custom grammars beyond JSON, regex-constrained fields, and the ability to combine masking with speculative decoding. It also hands you the operational risk of grammar compile time, cache behaviour and engine bugs. Treat it as an infrastructure dependency with its own regression tests.

Inside Constrained Decoding: Masks, Automata and Caches

Constrained decoding converts a schema into a machine that answers one question at every step: given what has been generated so far, which vocabulary tokens can legally come next? Those tokens keep their logits; every other token is set to negative infinity before sampling. Because masked tokens have probability zero, invalid output is not unlikely, it is unreachable.

Figure 2: Constrained decoding for LLM structured output. The schema is compiled once into an automaton or stack machine, a token mask is derived per parser state, applied to the logits, and the sampled token advances the parser.

The loop in Figure 2 has two halves that run in parallel. The model half is ordinary: prompt tokens go through a forward pass and produce logits. The grammar half owns a parser state, looks up or computes the legal-token mask for that state, and after sampling advances the parser by the chosen token. The two meet at the mask application step. The engineering challenge is making the grammar half faster than the model half so it never becomes the bottleneck.

Why it is hard: tokens are not characters

Grammars speak in characters; models speak in tokens. A single token can span several grammar symbols, for example a token that is a closing quote followed by a comma and a space. A naive implementation checks every one of the 100,000-plus vocabulary entries against the parser at every step, which is far too slow. The art is in avoiding that scan.

Two distinct strategies appear in the engines’ own documentation. XGrammar precomputes, for each grammar position, which tokens are valid regardless of the surrounding stack (context-independent) and caches them; only the context-dependent remainder is checked at runtime, using a persistent stack structure, while the work overlaps with GPU execution. llguidance instead parses with Earley’s algorithm over a lexer built from regular-expression derivatives and computes masks by walking a prefix tree (trie) of the tokenizer vocabulary, building automata lazily. Its authors claim roughly 50 microseconds of single-core CPU time per token for a 128k-vocabulary tokenizer and negligible startup, and these are the project’s own figures.

The practical trade is startup versus steady state. Heavy precomputation pays off when the same schema is reused across many requests and costs you on the first call. Lazy computation starts instantly and pays a little on each step. llguidance’s own comparison notes describe other engines as having large startup costs or seconds-to-minutes precomputation on some grammars; treat those as vendor-stated, and measure on your schema.

Forced tokens and fast-forwarding

A schema fixes much of the output text. Once the decoder has emitted {"customer_id":, the next characters of the following key are often fully determined. llguidance documents “fast-forward” tokens: where the grammar allows only one continuation, the engine can append the forced text without sampling it from the model, saving forward passes. The same observation underlies the speculative-decoding angle: deterministic scaffolding is the easiest thing to draft. See our guide to speculative decoding with EAGLE-3, Medusa and lookahead for how drafting interacts with constraints.

What masking cannot do

Masking guarantees syntax and the schema constructs the engine implements. It cannot guarantee that the model chooses sensible values. It also distorts the model’s distribution: by zeroing tokens, probability mass is renormalised over the survivors, and if the model “wanted” a token the schema forbids, it is pushed toward the next-most-likely legal one, which may be a worse answer. This is the mechanism behind the quality effects discussed later. A related trap is key ordering: because the decoder emits fields in schema order, putting an answer field before a reasoning field forces the model to commit before it thinks.

Provider Behaviour: What the Official Documentation Actually Says

Provider documentation is the only defensible source for limits, and it changes. The statements below are drawn from the vendors’ pages as read in October 2026; re-check before you depend on a specific number.

OpenAI

The Structured Outputs guide states that both JSON mode and Structured Outputs ensure valid JSON, but only Structured Outputs ensures schema adherence. It is enabled with json_schema and strict: true. The examples set every property as required and additionalProperties: false, which reflects the documented pattern of expressing optional fields as nullable unions rather than omitting them. Safety refusals are programmatically detectable via a refusal field on the message, so your code must branch on it rather than assume a parseable payload. The guide points to a separate page for the supported-schema subset; I could not confirm specific nesting or property-count limits from the main guide, so I will not quote any.

Anthropic

Anthropic’s documentation describes two complementary features: JSON outputs via output_config.format, and strict tool use. Several details are decision-relevant:

Objects must set additionalProperties: false.
Supported constructs include enums, const, anyOf, internal $ref and $defs, a set of string formats, and minItems of 0 or 1 only.
Unsupported constructs include recursive schemas, external $ref, numeric bounds such as minimum and maximum, string length bounds, and additionalProperties other than false. The SDK helpers can strip these and fold them into field descriptions, while still validating the original constraints client-side.
The first request with a new schema pays grammar-compilation latency, and compiled grammars are cached for 24 hours from last use. Changing schema structure invalidates the cache; changing only names or descriptions does not.
Complexity limits apply across all strict schemas in a request, including 20 strict tools, 24 optional parameters and 16 union-type parameters, plus a 180-second compile timeout and internal grammar-size limits that surface as a “too complex” error.
A refusal stop reason returns a 200 and may not match the schema. A max_tokens stop reason can leave truncated, invalid JSON.
Citations and prefilling are incompatible with JSON outputs. Enum capitalisation may differ from the schema in some cases, so compare case-insensitively.

Google Gemini

The Gemini structured-output page documents JSON Schema support for types including string, number, integer, boolean, object, array and null, with nullability expressed as a type array. Documented keywords include properties, required, additionalProperties, enum, format for date-like strings, numeric minimum and maximum, and minItems/maxItems. It states that minProperties, maxProperties, minLength and maxLength are not enforced during decoding; the server strips them rather than erroring, so you must validate them in application code. Recursive schemas via $ref are supported, streaming yields partial JSON chunks, and the page notes that very large or deeply nested schemas may be rejected.

What the three have in common

Read together, the pages share a pattern worth internalising. Every provider supports a subset of JSON Schema. Numeric and length bounds are the most commonly unsupported constraints, which is exactly where application logic usually lives. Refusals and truncation are the two ways a “guaranteed” mode still returns something unusable. And first-call compile latency is a real operational cost for dynamic schemas. Design for the intersection if you want portability across vendors, which in practice means flat-ish objects, required fields, enums for closed sets, and validation of bounds in your own code.

Measuring It Honestly: A Benchmark Harness and Validation Loop

Published leaderboards rarely transfer to your schema. JSONSchemaBench exists precisely because performance varies with schema complexity, and the only trustworthy number is one measured on your own schemas, your own prompts and your own model. This section describes what to measure, then gives a harness you can run. I am deliberately not printing a results table: any precise percentages I wrote here would be invented, and the brief for this site forbids that. Instead, the methodology tells you how to produce your own.

Five metrics, kept separate

Most blog benchmarks collapse everything into “accuracy”. Keep these apart:

Parse rate. The fraction of responses that json.loads accepts after you strip nothing. Prompt-only modes are tested here; enforced modes should sit at the ceiling unless truncation or refusal intervenes.
Schema-validity rate. The fraction that passes a full JSON Schema validator against the original, unmodified schema. This catches providers that silently drop unsupported keywords.
Semantic accuracy. Field-level agreement with gold labels, or a rubric-based judge for free-text fields. This is the metric constrained decoding does not guarantee. For judge design, see our notes on LLM-as-a-judge calibration and bias.
Latency decomposition. Time to first token, inter-token latency, and total, reported separately for the first request with a schema (cold compile) and subsequent ones (warm cache).
Retry-adjusted cost. Tokens billed across all attempts needed to reach a valid object, because a mode with a lower raw validity rate may still be cheaper if retries are rare and cheap.

Designing the test set

Use at least three schema tiers: a flat object of five to ten scalar fields, a nested object with arrays and enums, and an adversarial schema with unions, optional fields and deep nesting. Vary the input too: short clean text, long noisy text, and inputs that deliberately lack the information a required field wants, because that last case is where models invent values to satisfy the schema. Run each cell enough times to see variance (temperature above zero exposes more failures), and record the stop reason on every call, since truncation and refusal explain a large share of “enforced mode failed” reports.

The harness

The following script is provider-agnostic. You supply an async call_model function that returns the response text and a stop reason; everything else (validation, retry with error feedback, metrics) is shared. It uses the jsonschema library for validation, which implements the JSON Schema specification independently of any vendor.

import json, time, statistics
from dataclasses import dataclass, field
from jsonschema import Draft202012Validator

@dataclass
class Attempt:
    text: str
    stop_reason: str
    latency_s: float
    parsed: bool = False
    schema_valid: bool = False
    errors: list = field(default_factory=list)

def check(text, validator):
    """Return (parsed, schema_valid, errors) with no text cleanup."""
    try:
        obj = json.loads(text)
    except json.JSONDecodeError as e:
        return False, False, [f"parse: {e.msg} at {e.pos}"]
    errs = [f"{'/'.join(map(str, e.path)) or '$'}: {e.message}"
            for e in validator.iter_errors(obj)]
    return True, not errs, errs

def run_case(call_model, prompt, schema, max_attempts=3):
    validator = Draft202012Validator(schema)
    attempts, feedback = [], ""
    for _ in range(max_attempts):
        t0 = time.perf_counter()
        text, stop = call_model(prompt + feedback, schema)
        a = Attempt(text, stop, time.perf_counter() - t0)
        a.parsed, a.schema_valid, a.errors = check(text, validator)
        attempts.append(a)
        if a.schema_valid:
            break
        # Truncation needs a bigger budget, not a longer prompt.
        if stop == "max_tokens":
            feedback = "\n\nYour previous answer was cut off. Be more concise."
        else:
            feedback = ("\n\nYour previous answer failed validation:\n- "
                        + "\n- ".join(a.errors[:5]) + "\nReturn corrected JSON only.")
    return attempts

def summarize(cases):
    first = [c[0] for c in cases]
    n = len(first)
    return {
        "parse_rate": sum(a.parsed for a in first) / n,
        "schema_valid_rate": sum(a.schema_valid for a in first) / n,
        "final_valid_rate": sum(c[-1].schema_valid for c in cases) / n,
        "mean_attempts": statistics.mean(len(c) for c in cases),
        "p50_latency_s": statistics.median(a.latency_s for a in first),
        "p95_latency_s": sorted(a.latency_s for a in first)[int(0.95 * (n - 1))],
        "truncated": sum(a.stop_reason == "max_tokens" for a in first) / n,
    }

Notice what the harness refuses to do: it never strips markdown fences or trims prose before parsing. If you clean up output in the harness, you measure your cleaner, not the model. In production you may well want a tolerant parser as a fallback, but it belongs in a separate, labelled column.

Wiring a provider

The only provider-specific code is call_model. Below is a sketch for Anthropic’s documented output_config.format shape, taken from the vendor’s own example. The max_tokens value and model name are placeholders you must set; the structure follows the documentation.

import anthropic
client = anthropic.Anthropic()

def call_model(prompt, schema, model="YOUR_MODEL_ID"):
    resp = client.messages.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
        output_config={"format": {"type": "json_schema", "schema": schema}},
    )
    text = next((b.text for b in resp.content if b.type == "text"), "")
    return text, resp.stop_reason

For OpenAI the equivalent passes a json_schema object with strict: true in response_format or text.format, and you read refusal before reading content. For a self-hosted vLLM or SGLang server, you pass the schema through the server’s structured-output parameter and select the backend (XGrammar, llguidance) by flag. Because call_model hides all of that, the same cases run across every layer and you compare like for like.

The validation loop in sequence

The harness encodes a loop that is also the right production shape. The application sends a request with the schema, receives text plus a stop reason, validates syntax and schema, then validates business rules that JSON Schema cannot express, and retries with specific error feedback before failing closed.

Figure 3: The validate-and-retry sequence for LLM structured output. Syntax and schema checks run first, business-rule checks second, and feedback-driven retries are bounded before the request fails closed.

Two design points matter. First, the feedback to the model should be the validator’s actual error list, not a generic “try again”: specific messages like items/2/quantity: -1 is less than the minimum of 0 give the model something to fix. Second, bound the retries. A model confused about a schema usually stays confused, so the third attempt rarely succeeds where the second failed. After two or three attempts, route to a dead-letter queue or a fallback model rather than burning tokens.

Interpreting results without fooling yourself

When enforced modes show 100 percent schema validity, that is the expected outcome, not a finding. The interesting rows are elsewhere. Look at semantic accuracy, where enforced and unenforced modes can differ in either direction. Look at the cold-versus-warm latency gap, which tells you whether dynamic schemas are affordable. Look at the truncation column, which tells you whether your max_tokens is sized for the largest legitimate object. And look at cases where the input lacked required information, counting how often the model fabricated plausible values rather than emitting an explicit null or “unknown”.

A caution on statistics: with a few hundred cases per cell, differences of a point or two are noise. Report intervals, not point estimates, and do not declare a winner on a one-point gap. Also pin model versions and record them with each run, since providers update serving stacks and grammar engines without changing the model name you call.

Quality, Latency and the Cost of Constraints

The fear most teams raise first is that forcing a format hurts reasoning. The evidence is real but narrower than the fear suggests, and understanding the mechanism lets you avoid most of the damage.

What the research says

“Let Me Speak Freely?” (arXiv:2408.02442) compared format-restricted generation against free-form responses across common tasks and reported a significant decline in reasoning ability under format restrictions, with tighter constraints generally causing larger drops. That is a published finding worth taking seriously. Two caveats apply. It reflects the models, prompts and constraint implementations of its time, and later work and vendor engineering have contested how general the effect is. And the effect concentrates in reasoning-heavy tasks where the model is asked to think inside the constrained field. I would treat it as a design warning, not as a law: test your task.

The mechanism and the fix

The mechanism is mundane. If the first field of your schema is the final answer, the model must commit to it before producing any reasoning, and masking has already removed the free-text scaffolding it would normally use to think. The fix is structural. Put a free-text reasoning or analysis string field before the answer fields, so the model reasons in tokens and then fills the typed fields from that reasoning. Anthropic’s documentation adds a related caution: requests for step-by-step reasoning in the constrained output may trigger a reasoning-extraction refusal, and it recommends asking for a short explanation instead. Alternatively, use two calls: a free-form reasoning pass, then a cheap extraction pass into the schema.

Figure 4: Field ordering and stop-reason handling. Reasoning fields come first so the model thinks before committing; the stop reason routes truncation, refusal and validation failures to different handlers.

Latency: three separate costs

Constraint latency is not one number. There is compile latency, paid when a schema is first seen; Anthropic documents added latency on first use and a 24-hour cache keyed on schema structure. I could not confirm comparable figures for other hosted providers from their main guides, so benchmark it yourself. There is per-token mask latency, which self-hosted engines minimise through caching or lazy computation and which is negligible when overlapped with the GPU. And there is output length effect: strict schemas with long key names inflate output tokens, while forced scaffolding can reduce sampled tokens through fast-forwarding. Net latency can go up or down. Measure cold and warm separately.

A subtler cost: Anthropic notes that structured outputs add a system prompt, slightly increasing input tokens, and that changing the output format invalidates the prompt cache for that conversation thread. If your application varies schemas per request and relies on prompt caching, you can lose more than you gain. Pin schemas per workflow.

Schema design for quality and portability

Small schema choices have outsized effects. Use descriptive field names and description strings, which the vendors say guide the model. Prefer enums for closed sets, since they constrain the model to your vocabulary and make downstream branching safe. Make fields required and express absence as a nullable type, which matches the pattern in OpenAI’s own examples and keeps Anthropic’s optional-parameter budget free. Flatten where you can, because both Anthropic’s limits and Gemini’s warnings point at complexity as the thing that gets rejected. Avoid recursion if you target more than one vendor, since Anthropic’s documentation lists recursive schemas as unsupported while Gemini documents $ref recursion. When you need numeric bounds or string lengths, enforce them in code: at least two major vendors document that they do not constrain those during decoding.

Where tool calling still wins

Do not rewrite everything as response-format schemas. When the model must choose among actions, tool calling carries the right semantics: the tool name is the decision, the arguments are the payload, and the runtime executes it. This is also the substrate of the Model Context Protocol, where tool schemas are the interface contract between agents and servers. The practical rule: use a response schema when the reply is data; use strict tool calling when the reply is an action.

Trade-offs, Gotchas, and What Goes Wrong

Enforced structured output removes one class of failure and exposes others. These are the ones that bite in production.

Truncation produces invalid JSON even in strict mode. A decoder guarantees every emitted prefix is on a valid path; it cannot guarantee the output finishes. If the model hits max_tokens mid-object, you receive a syntactically incomplete document. Anthropic’s documentation names this explicitly via the max_tokens stop reason. Size the budget for your largest legitimate object, and always check the stop reason before parsing.

Refusals break the contract. When a safety refusal fires, the response may not match your schema. OpenAI exposes a refusal field; Anthropic returns a refusal stop reason with a 200 status that is still billed. Treat refusal as a first-class branch, not an exception to catch in a generic handler.

Schema-valid is not true. If a required field asks for a value absent from the input, the schema forces the model to produce something. Without an explicit escape hatch (a nullable type, an unknown enum member, or a boolean found flag), the model fabricates. This is the most common silent failure of enforced modes, and no amount of grammar engineering fixes it. Design escape hatches deliberately, and measure fabrication rate on inputs that omit information.

Silent keyword dropping. Vendors that strip unsupported constraints rather than rejecting them make your schema look stricter than it is. Gemini’s page says length and property-count bounds are stripped without an error; Anthropic’s SDK helpers move unsupported constraints into descriptions. Always validate the original schema client-side so a stripped constraint becomes a caught error.

Compile cost surprises. Schemas generated per request, for example from user-defined forms, defeat caching and pay cold-start compile cost every time. Anthropic documents a 24-hour cache from last use and a compile timeout; a pathological schema can fail outright with a complexity error. Cap schema size at the application layer and test your largest customer-defined shape.

Enum and format quirks. Documented enum capitalisation differences mean a downstream == against a schema constant can fail. Normalise before comparing. Avoid enum values that differ only by case.

Engine divergence when self-hosting. Different engines implement different subsets of JSON Schema, and JSONSchemaBench was built because coverage differs materially across them. Upgrading a serving image can change which schemas compile. Pin versions, keep a corpus of your production schemas, and run it in CI against every engine upgrade as a compile-and-sample smoke test.

Observability gaps. Structured-output failures are easy to hide behind a retry. Emit a span attribute for validation outcome, attempt count and stop reason so you can see a rising retry rate before it becomes an outage. Our guide to OpenTelemetry GenAI semantic conventions covers a common naming scheme for these signals.

Anti-patterns to avoid. Stripping fences and prose with regular expressions as the primary strategy (it hides a model problem and breaks on edge cases). Unbounded retry loops (they multiply cost during provider incidents). Giant monolithic schemas (they hit complexity limits and degrade quality; prefer several small calls). Putting the answer before the reasoning field. Assuming a vendor-specific feature behaves identically elsewhere.

Practical Recommendations

Start from your serving situation rather than from a feature list. If you call a hosted frontier model, use the provider’s schema-enforced mode for data replies and strict tool calling for actions, keep schemas flat and required, and validate bounds yourself. If you serve open-weight models, enable an XGrammar or llguidance backend through your inference engine, benchmark compile time on your real schemas, and keep a regression corpus. If you must support several vendors, design to the intersection of their documented subsets and put a thin adapter in front.

Then layer validation on top regardless of choice. Treat the decoder as eliminating syntax errors and the validator as catching semantics. Use specific error feedback in retries and cap attempts at two or three. For quality, put reasoning fields first, or split reasoning and extraction into two calls when the task is reasoning-heavy, and verify with your own evaluation rather than a published headline.

Use this checklist before shipping:

Schema uses required fields, nullable types for absence, and enums for closed sets.
Every object sets additionalProperties: false where the vendor requires it.
A reasoning or analysis field precedes answer fields for non-trivial tasks.
Stop reason is checked before parsing; truncation and refusal have explicit handlers.
Bounds, lengths, cross-field and business rules are enforced in application code against the original schema.
Retries are bounded, carry validator error messages, and end in a dead-letter queue or fallback.
Cold and warm latency are measured separately on your largest schema.
A fabrication test covers inputs that lack a required value.
Validation outcome, attempts and stop reason are emitted as telemetry.
Vendor documentation limits are re-checked at each model or SDK upgrade.

Frequently Asked Questions

What is the difference between JSON mode and structured outputs?

JSON mode guarantees the response parses as valid JSON but does not guarantee it matches any particular schema; keys, types and required fields can still be wrong. Structured outputs, in OpenAI’s terminology, add schema adherence on top, using a supplied JSON Schema with strict mode enabled. OpenAI’s documentation calls structured outputs the evolution of JSON mode and recommends it whenever possible. Other vendors use different names for the same idea, so check which guarantee you are actually getting.

Does constrained decoding hurt model quality?

It can, mainly on reasoning-heavy tasks where the model must think inside a constrained field. The study “Let Me Speak Freely?” reported a significant decline in reasoning under format restrictions, with tighter constraints hurting more. You can usually avoid the damage by placing a free-text reasoning field before the answer fields, or by splitting reasoning and extraction into separate calls. Test on your own task rather than assuming either outcome.

Can I get a 100 percent valid JSON guarantee?

For syntax and supported schema constructs, enforced modes get very close, because invalid tokens are masked before sampling. Two cases still break it: truncation when the token limit is reached, and refusals that override the schema. Also, vendors support only a subset of JSON Schema, so some constraints are not enforced. Check the stop reason, validate against your original schema, and keep a bounded retry path.

Which constrained decoding library should I self-host?

XGrammar and llguidance are the two engines most inference servers integrate; Outlines remains widely used. XGrammar emphasises precomputed token-mask caches and GPU overlap, while llguidance emphasises lazy automata and low startup cost. The right choice depends on your schema reuse pattern and serving engine. Benchmark compile time and per-token overhead on your own schemas, and pin versions because supported schema subsets differ.

Should I use tool calling or a response schema?

Use a response schema when the model’s reply is data you want in a fixed shape, such as extraction or classification. Use tool calling when the model must pick an action and supply arguments for your runtime to execute. Strict tool use, where available, validates tool names and inputs against the schema, so the two can be combined in a single request. The semantic distinction matters more than the mechanism.

How do I handle fields JSON Schema cannot constrain?

Validate them in application code. Several vendors document that numeric bounds, string lengths or property counts are not enforced during decoding, and cross-field rules like date ordering are outside JSON Schema entirely. Run a validator against the original schema after every response, add business-rule checks, and feed specific error messages into a bounded retry. Treat the decoder as handling syntax and your code as handling meaning.