Structured and schema-constrained generation.
Free text goes in. A record that always fits the shape comes out. Here is the field that makes a model speak data, not just prose.
A language model writes one word at a time, in prose. That is wonderful for talking to a person and a problem for talking to software. Structured generation is how the field closes that gap: it constrains what the model is allowed to say next, so the output is guaranteed to fit a required shape. We walk it the way the field is built: what it is, why it is hard, how it works, what it unlocks, and where it is going. The papers behind each idea are linked, so you can read the source yourself.
“Northwind, forty thousand. It hasn’t been approved yet.”
{
"company": "Northwind",
"amount": 40000,
"approved": false
}Making a model's output fit a required shape on purpose.
Not by luck and not by asking nicely. The shape is named up front as a schema or grammar, and the output is held to it.
A model writes prose for people. Software needs fields. Structured generation is the bridge that turns "writes for a human" into "returns data a program can read." You hand it a schema, a description of the valid form: these fields, these types, this layout. As the model writes, anything that would break the schema is simply not allowed.
Keep two jobs separate, because the rest of the page leans on the distinction. The shape is enforced by the system. The content, the actual values, is still the model's judgment. The constraint comes in degrees, from loose to strict: a regular expression is a pattern, a JSON schema names fields and types, a context-free grammar (Geng et al., 2023) can describe a whole formal language such as SQL. Across this page we follow one running example: pulling a clean record out of a messy sales call. The model reads the transcript. The schema guarantees the record comes back in a fixed, readable shape.
Schema
amount: number
currency: string
due_date: string or nullOutput
{
"amount": 40000,
"currency": "USD",
"due_date": null
}"Sounds right" and "parses" are not the same thing.
A fluent sentence is easy for the model and unusable for a program that expects exact fields. Closing that gap reliably is the whole problem.
A program needs exact fields.
"The deal is worth about forty thousand and closes next month" is fine for a person and useless to software. The receiving program needs a name, an amount, and a date, in an exact shape, or it breaks.
"Please reply in JSON" raises the odds, no more.
Asking for a format does not ensure it. One stray word, a trailing comma, an unclosed bracket, and the parser fails. At scale, "usually valid" still means a steady stream of breakages.
A record can be well-formed and still wrong.
Structure enforcement fixes the shape, not the truth of the values. Valid JSON does not mean a correct answer. We keep these as two separate guarantees so the page never overclaims.
More than checking brackets.
A JSON schema can require enums, nested objects, and required fields. A grammar can describe a whole formal language. Enforcing those exactly, token by token, is real work.
Checking the rules at every step adds up.
Validating each next token against the schema can add overhead unless the work is precomputed. This is a real engineering axis, not a footnote, and the field measures it.
A format can change the answer.
A growing body of work, including Tam et al. (2024), reports that how a format is imposed can affect reasoning quality on some tasks. Honest pages name this.
It all hangs on one choke point.
A model writes one token at a time, scoring every option in its vocabulary. Control what is allowed at each step, and you control the output's shape.
- 01Score
Model predicts candidate tokens.
- 02Constrain
Exclude invalid continuations.
- 03Select
Choose a valid token.
- 04Advance
Update the parser state and repeat.
Constrained decoding is a filter on the next word: block what breaks the shape, let the model choose the rest.
1The next-word filter
Hide the options that would break the shape.
Before the model picks, the system masks out every option that would violate the schema, setting those scores to effectively zero so they cannot be chosen. The model samples only from the survivors. The output stays valid by construction, not by hope.
A worked example. After the model has written {"amount": and the schema says amount is a number, the filter allows digit tokens and blocks letters, quotes, and brackets. The model still decides which number. The shape is not up for negotiation. That is the clean early lesson of PICARD (Scholak et al., 2021): constrain the decoder, not just the prompt.
{"amount":Allowed in this simplified numeric state
Blocked in this state
2How the shape is described
From a pattern to a whole language.
The constraint comes in degrees. A regular expression captures a simple pattern, a date or a phone number. A JSON schema names fields and types. A context-free grammar describes a full language such as SQL or a programming language. The richer the description, the richer the structure you can guarantee.
The unifying view from Geng et al. (2023): a wide range of structured tasks, extraction, disambiguation, parsing, can be expressed as "generate something that fits this grammar." That makes structure a single, reusable framework rather than a one-off trick per task.
| Constraint | What it describes |
|---|---|
| Pattern | A limited string shape, such as a date-like format. |
| JSON Schema | Object fields, types, and other supported constraints. |
| Grammar | Allowed syntactic structures of a language. |
3Turning a grammar into a fast filter
Valid output is a path through a set of states.
The reframing that made this practical: treat valid output as a path through a state machine. Willard and Louf (2023), the work behind the Outlines library, showed you can compile a regular expression or grammar into a finite-state machine and build an index over the model's vocabulary ahead of time. So at each step, the set of allowed tokens is a quick lookup rather than a fresh computation. Knowing the current state tells you exactly what is allowed next.
A careful filter aims to allow every valid continuation and block every invalid one. SynCode (Ugare et al., 2024) frames this directly: keep all tokens consistent with the grammar, drop the rest, so the result is both correct and not needlessly restrictive.
Allow valid numeric continuations or the permitted delimiter.
4Parsing as you go, and keeping it fast
Reject a wrong turn before it is taken.
For formal targets like SQL, an incremental parser can reject any token that could not begin a valid completion. PICARD did this for text-to-SQL, checking each step against the grammar so the model could not wander into code that would not run.
Making it fast enough to ship is its own line of work. Newer engines push the overhead down by splitting the vocabulary into tokens that can be checked ahead of time and the few that must be checked live, and by overlapping that work with the model's own computation. XGrammar (Dong et al., 2024) reports large speedups on this exact problem. That speedup is an industry finding from that paper, not a figure about any system described elsewhere on this site.
Prepare ahead
- Work
- Compile grammar state and reusable token checks.
Check per step
- Work
- Resolve state-dependent token validity as generation proceeds.
5A reliable tool call
A decision becomes a call a system can run.
When a model calls a tool, the call has to name the tool and fill its fields exactly. Structured generation is how "the model decided to look up an account" becomes a well-formed call a system can actually execute. The tool name and the field types are the schema. The constraint guarantees the call is shaped to run, the model still decides which account.
This is the backbone of assistants that do things, not just talk. The same machinery that returns clean JSON returns a clean function call.
{
"tool": "lookup_account",
"arguments": {
"company": "Northwind"
}
}- Validate
Check tool name and argument types.
- Authorize
Check whether this request may use the tool and data.
- Execute
Run the approved call and return its result.
Output software can read on the first try.
At the field level: clean records and rows, reliable tool and function calls, clean extraction from messy input, fewer retries, and interfaces a team can build on.
This family of techniques is how the industry gets dependable, machine-readable output out of a model that otherwise speaks in prose. For a sales co-pilot, the bridge is at the outcome level: here is what that reliability looks like as an everyday outcome for a seller. The deep mechanism stays on this educational page.
A call turns into clean fields.
The schema does the shaping and the model does the reading, so a transcript becomes named fields in the right places, not a wall of notes. See Clean CRM.
What lands in your tools is consistent.
A guaranteed shape is a contract. Records come back in a fixed layout, consistent enough to build a process on. See After-meeting notes.
Actions are real, not guesses.
When an assistant takes an action, it is a real, well-formed action, not a guess that fails halfway. Reliable tool calls are what make that possible.
Fewer retries, simpler plumbing.
When validity is guaranteed at generation time, you remove a whole layer of "parse, fail, ask again." That is less code and fewer failure modes downstream.
This explainer covers published research. To see how AI REV Voice helps with sales conversations, explore Prism.
Guaranteeing the shape is solved. Doing it without nudging the answer is the live work.
We describe the direction the field is heading, not a roadmap of our own.
Strict structure, essentially free.
The frontier is making guaranteed shape cost almost nothing at serving time, so there is no speed penalty for enforcing it. XGrammar and the surrounding engine work are the current direction.
Shape the result without bending the answer.
Studies such as Tam et al. (2024) report that how a format is imposed can affect reasoning quality. The response is constraints that let the model reason freely first and shape only the final result.
From flat JSON to whole languages.
From flat records toward deeply nested schemas, full grammars, and domain-specific languages, all enforced exactly, token by token.
Measure it, do not assume it.
Work like JSONSchemaBench (Geng et al., 2025) measures conformance, overhead, and quality on real schemas, turning "it usually works" into something you can test and compare.
Well-formed and more likely right.
The open frontier pairs guaranteed shape with checks on the content itself. Two separate guarantees: the structure is enforced, the truth of the values is a different question.
| Check | Example |
|---|---|
| Schema validity | amount is a number |
| Grounding | The amount appears in the source quote |
| Business validation | The currency and approval rules are satisfied |
The papers behind this page.
- Scholak, Schucher, and Bahdanau (2021). PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models. arXiv:2109.05093.
- Geng, Josifoski, Peyrard, and West (2023). Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning. arXiv:2305.13971.
- Willard and Louf (2023). Efficient Guided Generation for Large Language Models (Outlines). arXiv:2307.09702.
- Ugare et al. (2024). SynCode: LLM Generation with Grammar Augmentation. arXiv:2403.01632.
- Dong et al. (2024). XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. arXiv:2411.15100.
- Tam et al. (2024). Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models. arXiv:2408.02442.
- Geng et al. (2025). JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models. arXiv:2501.10868.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
That is the whole field in one promise: you talk, and clean, complete records are waiting in the right places. See it on a real call.
No app required. Just your phone number.