Structured Outputs and Tool Calling 
Making an LLM Return Something You Can Use 

The demo works because a human reads the output. The product does not, because a parser reads it. That is the whole difficulty in one sentence, and it is where most AI features spend their unplanned engineering time: the model produces something that is entirely correct as prose and unusable as data, with a stray sentence before the JSON, an enum value nobody defined, a date in a format the schema did not anticipate, or a perfectly formed object describing a record that does not exist. Getting a model to return something a program can rely on is a distinct engineering problem from getting it to be right, and treating them as one problem is why so many teams stall at eighty percent.

The four ways structured output is obtained, in ascending order of reliability

The weakest approach is asking politely in the prompt and parsing what comes back, which works often enough to be dangerous, since it will pass every manual test and fail a few percent of the time in production, usually on the inputs that matter most. A JSON mode, where the provider guarantees syntactically valid JSON, removes the parse failure but not the schema failure — you get valid JSON with the wrong shape, missing fields, or an invented enum. Constrained decoding against a schema is a substantially stronger guarantee, because the model is prevented at the token level from emitting anything that does not conform, and where a provider offers it, it should be the default. And the strongest available position is constrained decoding plus validation on your side, because a schema constrains shape but not meaning: it can guarantee a string field called status, and it cannot guarantee that the status is one your system recognises, that the referenced customer ID exists, or that the extracted total matches the sum of the line items. The rule worth adopting is that schema conformance is necessary and never sufficient — validate business invariants after parsing, in the same way you would for input arriving from any untrusted client, because that is precisely what this is.

Designing schemas the model can actually satisfy

• Keep them shallow. Deeply nested objects degrade quality noticeably; two or three flat calls almost always beat one elaborate one, and they fail in ways you can isolate.
• Use enums instead of free text everywhere a fixed set exists, and always include an explicit escape value — 'unknown' or 'other' — because without one the model will invent a plausible answer rather than admit uncertainty.
• Give every field a description in the schema. Providers pass these to the model, and they are frequently more effective than the equivalent instruction in the prompt.
• Make optional fields genuinely optional rather than requiring a null-filled object, and be explicit about what absence means.
• Ask for a confidence signal or a 'sufficient information' boolean on extraction tasks — models are imperfectly calibrated, but a model that says it could not find the value is far more useful than one that guesses.
• Request the source span or quote alongside any extracted value. It costs a few tokens, makes the output auditable, and turns a hallucinated field into something you can detect automatically.
• Avoid asking for numbers the model has to compute. Extract the components and do the arithmetic in code, always.

Tool calling: where it works and where it breaks

Tool calling shifts the problem from format to judgement. The mechanics are largely solved — the model reliably emits a well-formed call against a defined signature — and what remains unreliable is whether it chose the right tool, at the right time, with arguments that make sense given what it actually knows. The failure modes are consistent across systems. Models call a tool when they should have answered directly, or answer directly when they should have called a tool, and both are more common with a large tool set: past roughly ten to fifteen tools, selection accuracy falls off measurably, which is an argument for splitting agents by domain rather than expanding one agent's toolbox. They pass arguments that are syntactically valid and semantically invented, particularly identifiers, which is why every tool must validate its own inputs and return a useful error rather than trusting the caller. They loop, retrying a failing tool with slight variations until something stops them, so a hard limit on steps and on repeated identical calls is not optional. And they handle tool errors poorly unless the error message is written for them — a tool that returns 'ValidationError: 400' produces flailing, while one that returns 'no customer with that ID; use search_customers by name first' produces recovery. Write tool descriptions and error messages as though the consumer is a capable new engineer with no context, because functionally that is the situation.

The boundary that matters most

The most important design decision in a tool-calling system is which tools are allowed to change anything. A model choosing which read-only query to run is a manageable risk with a bounded downside. A model choosing to issue a refund, delete a record, send an email to a customer, or modify permissions is a different category of decision, and the fact that it is correct ninety-eight percent of the time is not reassurance — it is a statement that two percent of your write operations are wrong. The controls are ordinary engineering, not AI-specific: separate read tools from write tools and give the write ones narrow, specific signatures rather than a general 'update record' that accepts arbitrary fields. Enforce authorization inside the tool against the acting user's permissions, never in the prompt, because instructions are not a security boundary and anything reaching the model from a document, a webpage, or a customer message is a potential injection vector. Make write tools idempotent, since agents retry. Put a human confirmation in front of anything irreversible or externally visible, at least until you have production evidence that you do not need it. And log every tool call with its arguments and result, because this is the only way anyone will ever debug what the system did at three in the morning.

Handling the failures that remain

Even with constrained decoding and careful schemas, a percentage of calls will produce something unusable, and the difference between a robust feature and a fragile one is almost entirely in what happens next. Retry once with the validation error included in the context, since models correct their own schema violations at a high rate when told specifically what was wrong — but retry once, not indefinitely, and never with a higher temperature in the hope of a different outcome. Beyond that, degrade deliberately rather than erroring: return a partial result with the fields that validated, mark the task for human review, or fall back to a simpler extraction, depending on what the feature is for. Track the failure rate per schema as a first-class metric, because a drift from half a percent to four percent is usually the earliest signal that a provider has updated a model behind a stable name. And build the review path from the start — for anything consequential, a queue where a person sees the input, the output, and the reason it was flagged is worth more than another week of prompt tuning, and it doubles as the labelled dataset that improves your evaluation harness.

How Infiniti Tech Partners approaches this

Most of our AI engagements arrive at this problem within the first fortnight, because it is where the demo stops being a demo. The work is unromantic and largely determines whether the feature ships: schemas designed to be satisfiable rather than expressive, constrained decoding wherever the provider supports it, validation of business invariants after parsing, tool signatures narrow enough that a wrong call is cheap, and a read-write boundary with authorization enforced in code. We build the eval harness alongside it rather than after, because failure rate per schema is the number that tells you whether anything is improving. Where the feature touches money, customer communication, or permissions we default to a human confirmation step and remove it later on evidence, which is a conversation we would rather have at design time than after an incident. We keep a small number of concurrent engagements, so if you have an AI feature that works in review and misbehaves in production, this is usually a short, well-defined piece of work with a measurable outcome at the end.