Outlines

Last reviewed Aug 26, 2026 · 895 words

Outlines is a Python library, not a service, and the most useful thing to understand about it is what its guarantee covers: the output will parse. It will match your JSON schema, your regular expression, or your context-free grammar on the first try, with no retry loop, because Outlines masks every token the model is not allowed to produce next. What it does not guarantee is that the content inside the valid structure is correct. A field typed as an integer will be an integer; whether it is the right integer is still the model's problem.

Where it sits in a self-hosted AI stack

Self-hosters meet Outlines in two places. The first is indirectly: vLLM ships it as one of its guided-decoding backends, alongside XGrammar and lm-format-enforcer, so when you pass response_format with a JSON schema or guided_regex in a request to a self-hosted vLLM server, you may already be using Outlines' machinery without importing it. SGLang has its own equivalent. The second is directly: pip install outlines in a Python service that loads a model through Transformers, llama.cpp's Python bindings, or MLX on Apple silicon, and calls it with a Pydantic type as the output shape.

The direct route matters when you are writing the pipeline yourself, want regex or full-grammar constraints rather than only JSON, or need the same constrained decoding across a local model and a hosted API. For most people running Ollama behind a chat UI, the answer is simpler: Ollama's own format parameter accepts a JSON schema and enforces it through llama.cpp's grammar support, and llama.cpp accepts GBNF grammars directly. The wider practice of designing schemas and prompts for this is in structured output from LLMs.

The shape of the code

The API was reworked around the 1.0 release, so snippets copied from 2023-era blog posts will not run. The current pattern is to wrap a model from your chosen backend, then call it with a prompt and an output type:

import outlines
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer

class Ticket(BaseModel):
    category: str
    urgency: int
    summary: str

name = "Qwen/Qwen2.5-7B-Instruct"
model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained(name, device_map="auto"),
    AutoTokenizer.from_pretrained(name),
)

result = model("Classify this support email: ...", Ticket)
ticket = Ticket.model_validate_json(result)

The output type can be a Pydantic model, a JSON schema string, a Python Literal for multiple choice, a regex string, or a grammar. Constraining to Literal["billing", "bug", "feature"] is the cheapest trick in the library: three allowed continuations, no parsing, and a classifier that cannot answer "I'd say it's mostly a billing issue".

Why the first call is slow and the rest are not

The catalogue's "grammar compilation overhead" is real and worth planning around. Before generation, Outlines compiles your schema into a finite-state index over the tokenizer's vocabulary; for a complex schema on a 150,000-token vocabulary that takes seconds. The index is cached per schema and tokenizer, so a long-running service pays it once per schema and then decodes at close to unconstrained speed. A script that builds a fresh process per request pays it every time. Keep the model and its compiled schemas resident in one process, which is also the deployment shape a production Ollama or vLLM setup wants for other reasons.

The 8 GB figure is the model's, not the library's

Outlines itself is light; the listed 8,192 MB minimum is what a quantised 7B model needs to be loaded next to it. Run it against vLLM or llama.cpp over the network and the Python process holding Outlines can live on a 1 GB box. Run it in-process with Transformers and you need the VRAM or unified memory for the weights, plus a few hundred megabytes for the state machines. That framing is the decision: in-process for a single script or notebook, out-of-process against a shared inference server once 2 or more applications need the model.

Where the guarantee stops

Three things bite people after the first success. Schema validity does not stop the model from hallucinating plausible values into required fields, so make fields optional where "unknown" is a legitimate answer or you will get confident fabrication. Constraints on token choice can degrade quality slightly when the model wanted to write reasoning first; give it a reasoning string field ahead of the answer fields and quality returns. And a grammar for a full programming language is expensive enough that you should test compile time before committing to it. The tool-calling patterns in LLM tool calling sidestep most of these by keeping schemas small.

What I'd do

If your only need is JSON out of a local model behind Ollama or vLLM, use the runner's built-in schema enforcement and skip the dependency. Reach for Outlines directly when you are writing Python that owns the model, when you want regex or Literal constraints, or when you need one constrained-decoding interface across several backends. Keep schemas flat, put a reasoning field first, keep the process warm, and treat the output as syntactically trusted and semantically suspect.