Reliable JSON from a language model is a solved problem with a three-rung ladder: provider-native structured outputs where you have them, grammar-constrained decoding for self-hosted models, and a validate-and-retry loop as the universal fallback. If you're still regexing JSON out of markdown fences in 2026, you've skipped all three rungs. The unsolved part is subtler — schemas that force the model to hallucinate, and constraints that quietly degrade output quality — and that's where the design effort belongs.

Rung 1: native structured outputs

Major APIs now accept a JSON Schema and guarantee the response conforms — enforced at decode time, not "the model usually complies". Use it whenever available; malformed JSON simply stops being a failure class. Three practical caveats:

  • It's a schema subset. Typically: no recursive schemas, additionalProperties: false required on objects, and numeric/string constraints (minimum, maxLength, pattern) often unsupported — some SDKs strip them silently and validate client-side, others reject the schema. Enforce those constraints yourself after parsing.
  • First-call latency. New schemas usually pay a one-time compilation cost, then cache. Don't generate schemas dynamically per request.
  • Conformance is not correctness. The output will match the schema; whether the values are right is still your problem, which is why the validation rung below doesn't disappear.

Rung 2: constrained decoding for local models

Self-hosted models get the same guarantee through grammar-constrained decoding: the runtime masks the logits each step so tokens that would violate the grammar can't be sampled. Invalid JSON becomes impossible rather than unlikely.

  • llama.cpp takes GBNF grammars directly, and JSON Schema converts to GBNF — Ollama exposes this as a format parameter that accepts a schema.
  • vLLM ships structured-output backends (xgrammar and friends) that take JSON Schema, regex, or a grammar per request.

The throughput cost is modest — grammar compilation plus a small per-token overhead, low single-digit percent on current backends for typical schemas. Which server suits which workload is a separate question, covered in vLLM vs llama.cpp. Same caveat as rung 1, amplified for small models: the grammar guarantees syntax, and a 7B model under a tight grammar will happily produce perfectly-formed nonsense. Constraints don't add knowledge.

Rung 3: the validate-retry loop

The universal fallback, and the layer you keep even with rungs 1–2 in place, because it's where semantic validation lives:

from pydantic import BaseModel, ValidationError

class Verdict(BaseModel):
    evidence: list[str]
    confidence: float          # bounds checked below, not in the LLM schema
    verdict: str

def extract(text: str, retries: int = 2) -> Verdict:
    prompt = PROMPT.format(input=text)
    for _ in range(retries + 1):
        raw = llm(prompt, schema=Verdict.model_json_schema())
        try:
            v = Verdict.model_validate_json(raw)
            if not 0 <= v.confidence <= 1:
                raise ValidationError.from_exception_data("Verdict", [])
            return v
        except ValidationError as e:
            prompt = (f"{PROMPT.format(input=text)}\n"
                      f"Your previous output failed validation:\n{e}\n"
                      f"Return only corrected JSON.")
    raise ExtractionFailed(text)

The details that matter: feed the actual validation error back — Pydantic's messages are specific enough for the model to act on; cap retries at 2 (if it fails three times, the schema or prompt is wrong, and retry #7 won't fix it); and log the retry rate as a first-class metric. A retry rate above ~5% is a design smell, and it's the earliest drift signal you'll get when a provider updates a model underneath you.

Schema design the model can actually fill

  • Flat and boring wins. Nesting, oneOf unions, and polymorphism raise error rates on every rung. Two simple schemas beat one clever one.
  • Make "unknown" expressible. The classic failure: the schema requires "founded_year": integer, the input doesn't contain it, and the enforced decode must emit some integer — so you get a confident 1997 from nowhere. Nullable fields or an explicit "unknown" enum member aren't optional niceties; they're the difference between missing data and fabricated data.
  • Order fields so reasoning precedes conclusions. Generation is autoregressive: {"evidence": [...], "verdict": ...} lets the verdict condition on the evidence written before it. Putting the verdict first measurably hurts on judgment-type tasks.
  • Enums with an escape hatch. Closed sets are great until reality exceeds them; include "other" and log when it's used.
  • Descriptions on every field. They're instructions the model reads at exactly the right moment; the same discipline as tool schemas, which are the same machinery wearing a different hat.

Failure modes to design against

Truncation. max_tokens too low ends the output mid-object — syntactically invalid even under constrained decoding, since the constraint can't force the server to keep generating. Check the finish reason before parsing; "length" means raise the cap, not retry.

Over-constraining quality. Forcing complex reasoning directly into a rigid schema measurably degrades it — models reason better in prose than inside a JSON straitjacket. For hard tasks, either add a free-text scratch field ahead of the structured fields or run two passes: reason in text, then extract to schema with a cheap second call.

Enum coercion. Constrained to ["bug", "feature", "question"], every piece of spam becomes a "question". The wrong-but-valid answer is more dangerous than a parse error because nothing alerts on it — the pipeline stays green while the labels rot. Escape-hatch members plus a weekly look at the label distribution catch this; a category whose share doubles overnight is a schema bug wearing a trend costume.

Bottom line

Use the highest rung your stack supports, keep Pydantic-style semantic validation regardless, and spend your review time on the schema itself: nullable unknowns, reasoning-first field order, escape-hatch enums, and a retry-rate metric with an alert on it. Structure is enforceable now; truthfulness inside the structure is still yours to verify.