Valid JSON,
by construction.

An LLM that returns prose is a chatbot; an LLM that returns a guaranteed JSON object is a component. That's the leap structured outputs enable. When the model can reliably emit {"function": "get_weather", "city": "Paris"}, your code can parse it, call the function, and feed the result back — the loop that turns a language model into an agent that acts.

So structured outputs are the substrate of function calling and tool use. The whole edifice rests on one requirement: the output must conform to a shape your program expects, without exception. "Usually correct" isn't good enough when another program is the reader — a single malformed response is a crash, a retry, a bug. The engineering question is how to make conformance a guarantee rather than a hope, and the answer lives one level below the prompt.

The intuitive approach — instruct the model to "respond only with JSON matching this schema" — works surprisingly well and fails surprisingly badly. Prompting shifts the model's probabilities toward valid JSON, but it never sets the probability of invalid output to zero. So you still get occasional prose preambles ("Sure, here's the JSON:"), trailing commentary, wrong types (a number as a string), missing required fields, or a hallucinated key.

At small volume you shrug and retry. At scale a 1% malformed rate is catastrophic: one in a hundred requests breaks, retries cost money and latency, and the failures cluster on exactly the hard inputs you most need to handle. The root problem is architectural — a probabilistic generator asked to hit a discrete target will sometimes miss. You cannot prompt your way to a guarantee. What you can do is make the miss impossible by intervening during generation.

An LLM generates one token at a time: it produces a logit (a score) for every token in its vocabulary, turns those into probabilities with a softmax, and samples one. Constrained decoding inserts a step in the middle. Given the partial output so far and a grammar (or JSON schema), it computes the set of tokens that would keep the output valid, then masks every other token's logit to −∞ before the softmax.

After masking, the softmax assigns exactly zero probability to every illegal token, so the model cannot sample one — regardless of what it "wanted" to say. Repeat at every step and the completed output is guaranteed to parse. The grammar walks a little state machine (after { you must open a key; after a key you must write :; and so on), and the mask is recomputed each step from the current state. The model still chooses which valid token — the meaning is still its own — but it can never choose an invalid one.

Let the model's logit for token t be z_t, and let A be the set of tokens the grammar allows next. Masking replaces the logits, then softmaxes:

Because e^−∞ = 0, every disallowed token gets probability exactly 0, while the allowed tokens keep their relative odds and still sum to 1 — a valid distribution over only the legal choices.

The consequence is a hard guarantee, not a nudge:

The model's argmax might be an illegal token, but it can't be sampled — masking makes conformance certain. The runnable version below masks logits to a tiny JSON grammar and shows illegal tokens getting zero probability.

"Structured output" spans a spectrum of strength and cost:

Stronger guarantees cost a little: computing the allowed set each step adds decoding overhead, and a very tight schema can occasionally push the model toward awkward phrasings or truncated content when the only valid continuation isn't what it wanted to say. But for anything a program parses, schema enforcement is the right default — it deletes a whole class of bugs and most of your validation code. This is the backbone of tool protocols and every agent that calls functions.

Guaranteed valid is not guaranteed correct. Constrained decoding ensures the output parses and matches the schema — it says nothing about whether the values are right. The model can still emit a well-formed object with a wrong city, a hallucinated argument, or a plausible-but-false number. Schema enforcement removes syntax errors, not judgment errors, so you still validate semantics and handle bad values downstream.

Two subtler traps. Over-constraining: an overly rigid schema can hurt quality, because forcing the next token can cut the model off from a better completion or make it stop early — keep schemas as loose as your parser allows. And forgetting reasoning room: if you want the model to think before answering, give it a free-text field first and the structured fields after, or the constraint denies it the space to reason. Used with those cautions, structured outputs are what let you treat an LLM as a dependable part in a larger machine — the difference between a demo and a system.