05 Jan 2024 AI Engineering 5 min read

Structured Output: From Prose Parsing to Schema Contracts

JSON mode from OpenAI's DevDay 2023, function calling as a schema contract, grammar-constrained decoding with Outlines and llama.cpp, and validation-first pipelines built on Pydantic v2. A survey of structured output techniques as of January 2024: what each method guarantees, what none of them guarantee, and why structure beats parsing prose.

Why Parsing Prose Fails

Language models emit text. Applications consume data. Through most of 2023, teams bridged this gap with string operations: regular expressions over model output, stripping of Markdown code fences, prompts ending in "respond only with JSON". The approach survives a demo. It does not survive a pipeline.

The failure modes are mundane. A trailing sentence after the closing brace. Single quotes instead of double quotes. An unescaped newline inside a string value. If 2 percent of responses fail to parse, a nightly batch over 10,000 documents produces 200 unhandled errors. Prose parsing turns a probabilistic model into a probabilistic system boundary. That is the problem structured output addresses.

The instinctive fix — more prompt engineering — does not converge. Each model update shifts the failure distribution. Each new edge case adds another regex branch. By late 2023, the tooling had finally moved the problem to where it belongs: into the API surface and into the sampling loop itself.

A task arrives — the agent plans its next step. 1/4

JSON Mode Since DevDay 2023

On November 6, 2023, OpenAI announced JSON mode at its first DevDay. Setting the new response_format parameter to json_object instructs gpt-4-1106-preview or gpt-3.5-turbo-1106 to produce syntactically valid JSON. For the new GPT-3.5 Turbo, OpenAI reported a 38 percent improvement on internal format-following evals covering JSON, XML and YAML tasks.

The scope is narrow and documented. JSON mode guarantees syntax, not shape: the model returns a valid JSON object of arbitrary structure. The prompt must still explicitly ask for JSON. And a response truncated by the max_tokens limit is cut mid-object and no longer parses. JSON mode removes exactly one failure class. It defines no contract.

Function Calling as a Schema Contract

The stronger primitive is older. On June 13, 2023, OpenAI shipped function calling with gpt-4-0613 and gpt-3.5-turbo-0613. A developer describes functions as JSON Schema via the functions parameter; the model returns a JSON object whose arguments match the signature. The models were fine-tuned for exactly this mapping.

In practice, function calling doubles as an extraction API. Define a single function whose parameters are the target schema, force its selection via function_call, and the arguments string is your structured record. This yields schema-shaped output far more reliably than free prompting. It is still not enforced: required fields can be missing, enum values can be invented, and the arguments string itself can occasionally be malformed JSON.

Constrained Decoding at the Token Level

A third approach guarantees structure by construction. Grammar-constrained decoding masks, before every sampling step, all tokens that would violate the target format. llama.cpp merged grammar-based sampling with its GBNF format in July 2023. Willard and Louf formalised the technique in "Efficient Guided Generation for Large Language Models" (arXiv:2307.09702, July 2023): compiling a regular expression or context-free grammar into a finite-state machine index over the vocabulary reduces the per-token cost to O(1) on average. The open-source library Outlines implements this.

The trade-off is access. Constrained decoding requires the logits, so it works with self-hosted open-weight models, not behind hosted APIs. As of January 2024, API customers get JSON mode and function calling; hard grammar guarantees remain a privilege of self-hosting.

Method	Structural guarantee	Available since	Main limitation
Prompt instructions only	None	—	Fails silently and intermittently
JSON mode (OpenAI API)	Valid JSON syntax	Nov 6 2023	No schema; truncation breaks the object
Function calling (OpenAI API)	Schema-shaped via fine-tuning	Jun 13 2023	Not enforced; fields may be missing
Grammar-constrained decoding	Conformance by construction	Jul 2023 (llama.cpp, Outlines)	Requires logit access (self-hosted)

Validation First with Pydantic

None of the API-side mechanisms replace validation. We treat every model response as untrusted input, exactly like a request body arriving from the public internet. Pydantic v2, released June 30, 2023 with its validation core rewritten in Rust, validates 5 to 50 times faster than v1 — fast enough to sit in every request path without a measurable cost.

The pattern is symmetric. The Pydantic model is the single source of truth: model_json_schema() generates the schema that goes into the function definition or prompt; model_validate_json() checks the response at the boundary. Libraries such as Instructor package exactly this round trip over OpenAI function calling. One type definition, both directions.

The Repair Loop as Pipeline Design

A validation failure is not an exception to log. It is a signal to feed back. The ValidationError names the offending field and the violated constraint; appended to the conversation and re-requested, it lets the model correct most failures in a single attempt. We bound this loop at one or two retries, then route the record to a dead-letter queue for inspection.

This design makes output quality measurable. The first-pass validity rate per prompt and model version becomes a number you can log, alert on and compare across releases. When a model upgrade drops the rate from 97 to 89 percent, you see it the same day. Prose parsing offers no equivalent metric — only anecdotes.

What Structure Does Not Guarantee

Structure is syntax. Correctness is semantics. A schema-valid response can still contain a fabricated invoice number that matches the pattern, a plausible but wrong date, or a category picked arbitrarily from the enum. Validation catches type errors; it does not catch confident nonsense. Content evaluation remains a separate and largely unsolved discipline.

Constrained decoding carries its own caveat: forcing tokens into a grammar changes the output distribution. A model pressed into a rigid format mid-generation can produce degenerate completions the unconstrained model would not have chosen. Structure eliminates parse failures. It does not eliminate testing.

The Outlook for 2024

Our expectation for 2024: schema conformance becomes a first-class API parameter. The gap between JSON mode, which guarantees only syntax, and grammar-constrained decoding, which guarantees conformance but only self-hosted, is too visible to persist. We expect hosted APIs to accept a JSON Schema and guarantee that the response validates against it — enforced at decoding time, not through fine-tuning alone.

The deeper shift is architectural. When the typed model generates the schema, the prompt scaffolding and the validation, the type definition becomes the interface between application and language model — and the prompt becomes an implementation detail behind it. Ten years of API tooling taught us to design contract-first. 2024 is the year that discipline reaches LLM integration.