Structured Output Without Asking Nicely

Most companies that ship LLM features to customers don't ship a chatbot. They ship one button that does one predictable thing - summarize this document, draft this reply - because a free-text box is an invitation for users to ask the model anything, and an LLM answering anything is exactly the unpredictability most products can't afford.

Summarization is the easy version of this. Feed the model a clear instruction, ask for a summary under some word limit, and you'll get something usable most of the time. The trouble starts the moment your output needs structure a downstream system can actually parse - not just readable prose, but something a UI or a database can consume without a human checking it first.

Asking nicely works, until it doesn't

The obvious first move is to ask for the structure in the prompt. Tell the model to break the text into sections, wrap each one in tags, add a title and a link back to the source, and it will produce exactly that - most of the time. Point it at HTML-like tags instead of markdown, and it'll happily generate section, title, and link blocks that look like a small, well-formed document.

If LLMs behaved the way we're used to software behaving, that would be the end of the story. They don't. Ask the same well-specified question a hundred times and a few of those answers will skip a tag, nest something wrong, or drift out of the format entirely - not because the model misunderstood the instruction, but because it's a probabilistic process, not a parser. And in production, "wrong shape ten times out of a hundred" is a broken UI, a parser exception, or a write to a database with fields that don't exist.

The instinct is to just retry until you get a clean parse. It works, but it costs tokens and time on every failure, and providers increasingly rate-limit you hard enough that a retry loop isn't free insurance - it's a queue.

Making the shape non-negotiable

The fix isn't a better prompt. It's moving the constraint out of the prompt and into the decoding process itself.

Tool calling is the version of this most people reach for first. You define a schema - a function name, a set of typed parameters, which ones are required - and pass it alongside the request. The model doesn't just try to write JSON that happens to match your schema; the API constrains which tokens it's even allowed to emit next, so the output is validated against the schema as it's generated, not after the fact.

tool = {
    "name": "create_section",
    "description": "Create a blog section with title, body, and link",
    "parameters": {
        "type": "object",
        "properties": {
            "title": {"type": "string"},
            "body": {"type": "string"},
            "link": {"type": "string", "format": "uri"},
        },
        "required": ["title", "body", "link"],
    },
}

Call the model with that tool attached, and what comes back is a structured argument object, not a string you have to hope parses cleanly.

Under the hood, this works because the model generates one token at a time, and a schema can be compiled into a constraint on which tokens are legal at each position - a finite state machine, effectively, sitting between the model's raw probabilities and the token it actually emits. A token with a high probability but the wrong type gets excluded outright; whatever's left that satisfies the schema is what gets picked from. That's the same mechanism behind OpenAI's structured json_schema response format - the difference is mostly where the payload ends up in the response, not how the enforcement works.

The point isn't the schema, it's giving up on hoping

None of this requires the model to get smarter. It requires you to stop treating format compliance as something you request and start treating it as something you enforce at the API boundary - the same instinct that makes you validate input at a REST endpoint instead of trusting the client to send well-formed JSON.

If you're still writing a regex to salvage a slightly malformed response, or building a retry loop and calling it resilience, that's usually a sign the constraint belongs one layer down, in the schema, not in the words you used to ask nicely.