SPEC: Structured Output

Definition

[Definition] Structured output is the ability to constrain an LLM to produce output that conforms to a specific format or schema — most commonly JSON, but also XML, CSV, or custom formats. It enables reliable programmatic consumption of LLM responses without fragile string parsing.

The Problem with Free-Form Output

Without structured output, LLMs produce conversational text:

Extracting data from this with code is brittle — the model might say "thirty two" or "NYC" next time.

With structured output:

{"name": "John Smith", "age": 32, "city": "New York"}

Guaranteed parseable. Same every time.

Methods for Getting Structured Output

1. Prompt-Based (Unreliable)

Just ask nicely:

Problem: model still sometimes adds preamble ("Here is the JSON: ..."), wraps in markdown code blocks, or deviates from the schema. Requires validation + retry logic.

2. JSON Mode (API-Level Enforcement)

Most providers offer a JSON mode that guarantees valid JSON:

# OpenAI
response = client.chat.completions.create(
    model="gpt-4o",
    response_format={"type": "json_object"},
    messages=[...]
)

Limitation: guarantees valid JSON but NOT adherence to a specific schema.

3. Structured Outputs / JSON Schema (Strongest)

Specify the exact schema the model must follow:

# OpenAI structured outputs
from pydantic import BaseModel

class Customer(BaseModel):
    name: str
    age: int
    city: str

response = client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=[...],
    response_format=Customer
)
customer = response.choices[0].message.parsed  # Typed Python object

[Key Insight] Uses constrained decoding (grammar sampling) to guarantee schema adherence at the token level.

4. Constrained Decoding (Outlines / LMQL)

At the sampling level, restrict which tokens can be generated based on the current state of the schema:

• ▸Using a grammar/regex/JSON schema as a constraint
• ▸The model physically cannot generate tokens that would violate the schema
• ▸Libraries: Outlines, LMQL, Guidance, llama.cpp grammar mode

5. Tool Use for Extraction

Frame the structured output task as a tool call:

{
  "name": "save_customer",
  "description": "Save extracted customer information",
  "parameters": {"type": "object", "properties": {"name": ..., "age": ...}}
}

The model is forced to "call" the tool with structured arguments.

Schema Types Supported

Common Structured Output Use Cases

Data Extraction

{
  "entities": [
    {"text": "Apple", "type": "company"},
    {"text": "Tim Cook", "type": "person"}
  ],
  "sentiment": "positive",
  "summary": "..."
}

Classification with Confidence

{
  "category": "billing",
  "subcategory": "refund_request",
  "urgency": "high",
  "confidence": 0.92
}

Multi-Step Reasoning Output

{
  "reasoning_steps": ["step 1...", "step 2...", "step 3..."],
  "final_answer": "42",
  "confidence": "high"
}

Tool Call Parameters

{
  "tool_name": "search_web",
  "parameters": {"query": "...", "max_results": 5}
}

Schema Design Best Practices

1. ▸Keep it flat when possible — deeply nested schemas confuse models more
2. ▸Use descriptive field names — models use field names as semantic hints
3. ▸Add descriptions — JSON Schema description fields guide the model
4. ▸Use enums for constrained values — prevents creative interpretations
5. ▸Mark optional fields — required vs. optional matters for completeness
6. ▸Test with edge cases — null values, empty lists, ambiguous inputs

Provider Support (2024)

Reliability Hierarchy (Best → Worst)

Related Concepts

• ▸Tool Use, Prompt Engineering, Inference, Validation, Pydantic, API, Agent