JSON mode guarantees the output parses; strict structured outputs guarantee it matches your schema. Those are different promises. JSON mode still returns valid JSON with missing fields and invented keys. Constrained decoding removes that class of failure entirely, but only for structure, never for correctness.
What changed is that schema conformance moved from a prompting problem to a decoding problem. Once providers began constraining the sampler itself rather than asking the model nicely, the entire category of malformed-output retry logic became obsolete.
Plenty of production code still carries that retry logic, and plenty of teams still believe JSON mode does something it does not.
| Common belief | What is actually true |
|---|---|
| JSON mode enforces my schema | It guarantees syntactic validity only. Fields can be missing, extra, or renamed. |
| Structured outputs guarantee correct data | They guarantee shape. A correctly typed field can still hold a wrong value. |
| I can use any JSON Schema | Providers support a subset. Unsupported keywords are rejected or ignored. |
| Optional fields work as in normal JSON Schema | Strict modes typically require every property, with optionality expressed as a union with null. |
| Constrained decoding is free | It removes retries but adds schema compilation and can distort token probabilities. |
| Validation is no longer needed | Refusals, truncation and semantic errors all still occur. Validate anyway. |
How does constrained decoding actually work?
At each generation step, the model produces a probability distribution over the whole vocabulary. Constrained decoding masks every token that could not legally continue a valid document under your schema, then samples from what remains.
Because an illegal token can never be selected, the output cannot violate the grammar. This is a structural guarantee, not a statistical one.
Figure 1 — Where each approach intervenes
- PROMPTINGAsk nicelyInstruction says “return JSON”. Model usually complies. No guarantee at all.
- JSON MODEConstrain to valid JSONOutput is guaranteed parseable. Keys and types are not guaranteed.
- STRICT SCHEMAConstrain to your grammarSchema compiled to a state machine. Illegal tokens masked at every step.
- STILL YOURSSemantic validationIs the value right? No decoder can answer that.
Each layer removes a failure class and none removes the next one. The final layer never goes away, which is the point most implementations miss when they delete their validators.
The technique is well documented in the literature on guided generation, notably Willard and Louf’s work on translating schemas into finite state machines that index the vocabulary efficiently (arXiv:2307.09702).
Why does the schema subset trip people up?
Because a schema that validates perfectly in your test suite can be rejected by the API, and the reasons are not obvious.
Compiling a schema into a decoding state machine requires the schema to be finite and statically analysable. Keywords that imply arbitrary validation logic cannot be expressed as token masks.
| Construct | Typical strict-mode status | What to do instead |
|---|---|---|
| Optional properties | Not supported; all properties required | Type the field as a union with null |
| additionalProperties | Must be false | Enumerate every key explicitly |
| Numeric ranges | Commonly unsupported | Validate after decoding, or use an enum |
| String patterns | Commonly unsupported | Post-validate with a regex |
| Deep recursion | Depth-limited | Flatten the structure |
| Very large enums | Size-limited | Retrieve candidates first, then constrain |
The rule underneath all of these: the decoder can enforce shape, not semantics. Anything requiring the model to reason about a value rather than its type belongs in your validation layer.
What still breaks when the schema is guaranteed?
Four things, and none of them are fixed by constrained decoding.
Refusals
A model declining a request produces a refusal, not your object. Code that assumes a schema-shaped response will throw on the one path you most needed to handle gracefully. Check the refusal channel before parsing.
Truncation
Hitting the max token limit mid-object yields incomplete output. Constrained decoding keeps the document legal as far as it goes; it cannot conjure the remaining tokens. Always inspect the finish reason.
Semantically valid nonsense
A field typed as a date will contain a well-formed date. Whether it is the right date is entirely outside the guarantee.
This is the failure that survives longest in production, because everything downstream is type-safe and nothing complains.
Enum coercion
When the true answer is not in your enum, the model must still pick one of your values. Constraining the output guarantees a legal value and quietly forces a wrong one.
Watch for this
Always include an explicit “unknown” or “other” member in any classification enum. Without it, the decoder cannot express uncertainty and will silently produce your closest available label. This single omission causes more bad data than every parsing bug combined.
How should you design the schema itself?
Schema design affects accuracy, not just structure, because field names and ordering are read by the model as instructions.
- Name fields descriptively. A key called customer_sentiment_score_negative_to_positive outperforms one called score, because the name is the only instruction the model gets at that position.
- Order fields so reasoning precedes conclusions. Generation is left to right, so a reasoning field placed before verdict lets the model condition its answer on its own analysis.
- Prefer enums to free strings anywhere the value space is genuinely closed, then add the escape hatch described above.
- Keep nesting shallow. Deep structures raise compilation cost and give the model more chances to lose the thread.
- Use descriptions as micro-prompts. Schema descriptions are visible to the model and are the cheapest place to disambiguate a field.
The ordering point is the one that surprises people. Putting the conclusion first forces the model to commit before it has reasoned, and measurably degrades quality on anything requiring judgment.
What does a defensible pipeline look like?
Figure 2 — Handling order, and what each branch means
- Yes
Handle as a policy outcome.Never parse. Surface to the user or route for review. - No
Continue to the finish reason.Do not assume completeness yet
- Yes
Treat as failure and retry with a larger budget.Partial objects poison downstream systems - No
Structure is now trustworthy.Shape guaranteed by the decoder
- Yes
Accept.Ranges, dates, referential integrity all checked - No
Quarantine, do not retry blindly.Repeated semantic failure usually means a schema or prompt defect
Note what is missing: there is no JSON parse-error branch. With strict schema constraints that branch is unreachable, and keeping dead retry logic around it is how codebases accumulate misleading complexity.
Where do teams go wrong?
Deleting validation because the schema is guaranteed
The guarantee covers shape. Business rules, cross-field consistency and referential integrity are all still yours. Teams that remove their validators after enabling strict mode trade loud parse failures for silent data corruption.
Retrying semantic failures
A retry fixes transient problems. When a model repeatedly returns the wrong value in a correctly typed field, retrying burns tokens and changes nothing. That signature almost always means an ambiguous field name, a missing enum member, or a genuinely underspecified task.
Treating one giant schema as good design
A schema with forty fields asks the model to hold forty concurrent obligations. Splitting into two focused calls usually improves accuracy on both, and the cost difference is smaller than the quality difference.
What do experienced teams do differently?
They version schemas like database migrations.
A schema change alters model behaviour, so it belongs in version control with a migration note, and outputs are stored with the schema version that produced them. Without that, debugging a data quality question from three months ago is guesswork.
They also log the raw response alongside the parsed object. When a semantic error surfaces later, the raw text frequently shows the model hedging or contradicting itself in a way the parsed object erased.
A short glossary
- Constrained decoding
- Masking tokens that cannot legally continue a valid document at each generation step, making violations impossible rather than unlikely.
- JSON mode
- A setting guaranteeing syntactically valid JSON output, with no guarantee about keys, types or structure.
- Strict schema mode
- Compiling a supplied JSON Schema into a decoding constraint so output is guaranteed to conform to it.
- Refusal channel
- A separate response path used when a model declines a request, which does not conform to the requested schema.
- Enum coercion
- The forced selection of a legal enum value when the correct answer is not among the permitted options.
Key takeaways
- JSON mode guarantees parseability; strict schema mode guarantees conformance. They are different promises.
- Constrained decoding masks illegal tokens at each step, making schema violations structurally impossible.
- Provider schema support is a subset: expect all-properties-required, additionalProperties false, and no numeric ranges or string patterns.
- Refusals, truncation, semantic errors and enum coercion all survive the guarantee.
- Always include an “unknown” member in classification enums so the model can express uncertainty.
- Order fields so reasoning precedes conclusions, since generation runs left to right.
- Above all, do not delete your validation layer — shape correctness is not value correctness.
Frequently asked questions
What is the difference between JSON mode and structured outputs?
JSON mode guarantees the response is syntactically valid JSON that will parse without error. Structured outputs additionally guarantee the response conforms to a schema you supply, with the required keys, types and nesting. JSON mode can return valid JSON with missing or invented fields; strict schema mode cannot.
Does constrained decoding reduce output quality?
It can shift behaviour, because masking tokens redistributes probability across the remaining options. In practice the effect is small compared with the reliability gained, but it is a reason to keep quality evaluations running after enabling strict mode rather than assuming the change is purely additive.
Why is my JSON Schema rejected by the API?
Providers support a subset of JSON Schema that can be compiled into a decoding state machine. Common causes are optional properties, additionalProperties not set to false, numeric range constraints, string patterns, and excessive nesting depth. Express optionality as a union with null and validate ranges after decoding.
Do I still need validation with structured outputs?
Yes. The guarantee covers structure, not meaning. A correctly typed date field can hold the wrong date, an enum can be coerced to a wrong-but-legal value, and refusals and truncation still occur. Keep business-rule validation and treat the schema guarantee as removing only parse failures.
How do I handle optional fields in strict mode?
Type the field as a union of its real type and null, and mark it required. The model then always emits the key and supplies null when there is no value. This preserves the all-properties-required constraint while giving you the same information an optional field would have.
What happens if the right answer is not in my enum?
The model is forced to choose one of your permitted values, producing a legal but wrong result with no error raised. This is why every classification enum should include an explicit “unknown” or “other” member, which lets the model signal that none of the options fit.
Should reasoning fields come before or after the answer?
Before. Generation proceeds left to right, so a reasoning field placed ahead of the conclusion lets the model condition its answer on its own analysis. Putting the verdict first forces a commitment before any reasoning has been produced, which degrades quality on judgment tasks.
Can I use structured outputs with streaming?
Generally yes, but partial output is not a valid object until generation completes. Treat streamed fragments as display-only and never feed them into downstream systems until the finish reason confirms the response ended normally rather than hitting a token limit.
References
- Willard, B. T., and Louf, R. Efficient Guided Generation for Large Language Models. arXiv:2307.09702. Describes compiling schemas and regular expressions into finite state machines for constrained sampling.
- JSON Schema. JSON Schema specification.
- OpenAI. Structured Outputs guide, covering strict schema mode, the supported schema subset, and refusal handling.
