Structured Outputs for LLM Apps: Get Reliable JSON Every Time

Structured Outputs for LLM Apps: Get Reliable JSON Every Time — cover image

Every team that ships its first AI feature meets the same quiet problem. The demo works beautifully, then a week into real usage the model answers with a friendly sentence instead of the JSON your code expected, and an order form, a CRM update or a report silently breaks. The fix is not a cleverer prompt. It is a discipline called structured outputs: telling the model exactly what shape its answer must take, and checking that it did.

In this guide we explain how structured outputs work, how to design schemas that models follow well, how to validate and recover from bad responses, and how to roll it out step by step in a real product. It is written for founders and product owners as much as developers, because the decisions you make here affect reliability, support load and cost.

Why free text breaks real products

A language model is built to produce fluent text. When you ask it to "return the customer name, the issue type and the urgency", it will often comply, but it may also add a greeting, wrap the answer in a code block, rename a field, or invent a new urgency level such as "pretty high". Each of these small changes is harmless to a human reader and fatal to a program that expects exact keys.

The problem grows with scale. For example, a feature that handles 200 requests a day might see only a handful of malformed answers, which feels tolerable. At 20,000 requests a day, the same rate could mean dozens of broken records every day, each one a support ticket or a wrong invoice. Reliability work done early is far cheaper than cleanup done later.

A prompt is a request. A schema is a contract. Production software needs contracts.

What structured outputs actually are

Structured outputs are a family of techniques that make a model respond in a predictable format. The common building blocks are:

If you already use shared validation on your web forms, the idea will feel familiar. Our article on shared validation schemas with Zod across frontend and backend shows how one schema can protect every layer of an app. The same schema can describe what the model must return.

Real-world example: support ticket triage

Imagine a service business that receives support emails. The goal is to classify each email, extract the order number, detect urgency and draft a short reply for an agent to review. Without structure, the model returns a paragraph of analysis. With structure, it returns this:

Now the helpdesk can route the ticket automatically, show the order in a side panel, and sort the queue by urgency. When the model is unsure, it sets needs_human to true and a person takes over. This pattern is also the backbone of the workflows described in our guide to human in the loop approval workflows for AI agents.

On a project like this, an illustrative outcome could be that agents spend far less time reading and sorting, and more time resolving. The exact gain depends on volume and how messy the inbox is, so measure it on your own data before promising a number to anyone.

Designing schemas models follow well

Schema design is where most of the quality is won or lost. A few rules that consistently help:

Keep schemas small and flat

Deeply nested objects and long lists of optional fields confuse models and make validation errors hard to read. Prefer a flat structure with clear names. If you need a complex result, split the task into two calls, each with a simple schema.

Use enums for anything categorical

Whenever a field has a fixed set of possible values, list them. Free text for categories leads to "Billing", "billing issue" and "payments" all meaning the same thing, which wrecks your reports.

Name fields like instructions

Models read field names and descriptions as part of the prompt. A field called "urgency" with the description "high only if the customer mentions a deadline, outage or financial loss" performs better than a bare "priority" field. Descriptions are cheap prompt engineering that stays attached to the schema.

Allow an honest way out

Give the model permission to say it does not know. Nullable fields, an "unknown" enum value and a confidence or needs_review flag reduce the pressure to invent an answer. Hallucinated values that fit the schema are more dangerous than a visible null.

Version your schemas

Treat the schema like an API. When you change a field, bump a version so older stored records can still be read. This sounds heavy but saves painful migrations later.

Validation: trust but verify

Even with schema-constrained decoding, you should validate on your side. A schema ensures the shape, but you also need rules that reflect your business:

Lookup checks are especially valuable. If the model extracts an order number that does not exist, you have caught a hallucination automatically and can route the item to a person. Teams that skip this step often discover problems only when a customer complains.

A numbered process to add structured outputs to your product

  1. Pick one narrow task. Choose a single feature such as classifying inbound messages or extracting invoice fields. Resist the urge to convert everything at once.
  2. Write the schema first. List the exact fields, types and allowed values before touching the prompt. Review it with the person who will use the output.
  3. Collect 30 to 50 real examples. Use anonymised, real inputs with the correct expected outputs. This becomes your first test set. Our article on building LLM evaluation test sets for business AI features explains how to grow it over time.
  4. Turn on schema-constrained output. Use your provider's structured output or tool calling feature rather than asking politely in the prompt.
  5. Add a validation layer. Parse with a typed library, add business rules, and log every failure with the raw response.
  6. Implement retry and fallback. On a failure, retry once with the validation error included in the message. If it still fails, use a safe default or send to a review queue.
  7. Measure and monitor. Track the validation pass rate, retry rate and human override rate. A rising override rate is an early warning that your inputs or the model have drifted.
  8. Expand gradually. Once one feature is stable, reuse the same pattern for the next.

Handling failures gracefully

Plan for failure from day one. A good recovery strategy has three layers. First, a quick automatic retry that feeds the validation error back to the model, which often fixes small mistakes. Second, a deterministic fallback, such as a default category of "other" with needs_human set to true. Third, a review queue where a person fixes the record and, ideally, the corrected example is added to your test set.

Observability matters here as well. When something fails, you need the input, the schema version, the raw output and the error in one place. The practices covered in our guide to AI agent observability, tracing and debugging apply directly.

Common mistakes to avoid

Key benefits for your business

When to get expert help

Simple extraction tasks are well within reach of a capable in-house developer. The work becomes harder when you combine several models, connect outputs to payments or inventory, or need audit trails for compliance. That is where an experienced partner pays for itself. If you are planning an AI feature and want a second opinion on architecture, see how our AI development services approach reliable, production-ready builds.

Conclusion

Structured outputs turn an impressive demo into dependable software. The recipe is simple: define a small, clear schema, let the model fill it using native structured output features, validate everything against your business rules, and plan for the cases where it cannot. Add monitoring and a growing test set, and your AI feature becomes something your team can maintain with confidence rather than fear.

Start with one workflow this week. Write the schema, gather a few dozen real examples, and see how much cleaner your results become. Reliability is rarely glamorous, but it is the difference between an AI experiment and a product customers trust.

Frequently Asked Questions

What are structured outputs in an LLM app?
Structured outputs mean the model returns data in a fixed shape, usually JSON that matches a schema you define. Your code can then read fields like category or amount directly instead of parsing free text.
Is a JSON schema enough to guarantee correct answers?
No. A schema guarantees the shape of the response, not the truth of its content. You still need validation rules, business checks and evaluation test sets to catch wrong values.
Should I use function calling or a response format option?
Use a response format with a schema when you only need data back. Use function calling when the model must choose between actions or tools. Many apps use both in different places.
What should happen when the model returns invalid output?
Validate every response, retry once with the error message included, and fall back to a safe default or a human review queue if it still fails. Never pass unvalidated output into a database or payment flow.
Does this work with any AI model?
Most major model providers now support some form of schema-constrained output or tool calling. Where support is weaker, a validation and retry layer gives you similar reliability at a small extra cost.