Structured outputs: getting reliable JSON from an LLM
An LLM is fundamentally a text generator, and "asking nicely" for valid JSON in the prompt works right up until it doesn't. Structured outputs are the mechanism that makes the model's response conform to a schema by construction, not by good behavior.
3 min read
Why "just ask for JSON in the prompt" fails at scale
A prompt like "respond only with valid JSON matching this shape" works most of the time — which is exactly the problem. "Most of the time" means occasional malformed JSON (a trailing comma, an extra sentence of preamble before the {, a field renamed to something close-but-not-exact), and at any real volume, "occasional" becomes "several times a day," each one a runtime exception in code that assumed JSON.parse() would just work. This isn't a prompting skill issue — it's the model doing what generative text models do: producing plausible text, not validated data.
What "structured outputs" actually means
Providers now offer a mechanism where the response is constrained during generation — not validated after the fact, but literally prevented from generating a token that would violate the schema in the first place. This is fundamentally different from prompt-based instruction:
The most common implementation is via tool/function calling: define a "tool" whose parameters schema is the actual shape you want back, force the model to call it, and read the arguments it passed — the model was never really "using a tool," the tool-call mechanism is being repurposed as a structured-output channel because it already has the schema-constraint machinery built in.
A real example
// Vercel AI SDK — generateObject forces schema-conformant output
import { generateObject } from "ai";
import { z } from "zod";
const schema = z.object({
sentiment: z.enum(["positive", "neutral", "negative"]),
summary: z.string(),
confidence: z.number().min(0).max(1),
});
const { object } = await generateObject({
model,
schema,
prompt: `Analyze this review: ${reviewText}`,
});
// object is guaranteed to match `schema` — no parsing, no validation step neededThe schema is doing double duty: it's a contract for the model's output, and it's the same validation library (zod, here) your application code already uses for everything else — one schema, not a hand-maintained parallel description in a prompt and a separate TypeScript type that can drift apart.
When you still need application-level validation anyway
Schema-constrained generation guarantees shape — the right fields, the right types. It doesn't guarantee correctness — a confidence: 0.95 field constrained to be a number between 0 and 1 can still be a number the model made up with no real basis. Structured outputs solve the parsing problem, not the grounding problem; a hallucinated value that happens to be well-typed is still hallucinated. Treat schema conformance as a floor, not a substitute for validating the actual content when it matters (a proposed price, a decided-upon date, an entity ID that should exist in your database).
Further reading
Check your understanding
A quick comprehension check — not tracked, not graded, just for you.
1. A prompt instructs the model to "respond only with valid JSON." Why does this fail at real production volume even though it usually works?
2. How does schema-constrained generation differ from asking the model nicely for JSON in the prompt?
3. Why is tool/function calling commonly repurposed as a structured-output mechanism, even when no real tool is being executed?
4. A generateObject call returns an object matching its zod schema exactly, including a confidence field of 0.95. Does schema conformance guarantee that 0.95 is a meaningful, well-grounded value?