Guidance

Last reviewed Aug 26, 2026 · 911 words

Guidance is not something you deploy. It is a Python library (pip install guidance) that sits between your code and a language model and takes over the decoding loop, so the model can only emit tokens that fit the shape you declared: a regex, a JSON schema, a choice from a list, a full context-free grammar. Against a model loaded in the same process, that guarantee is absolute, because Guidance masks the logits before every token is sampled. Against a hosted API it degrades to careful prompting plus parsing, and that one fact decides whether it belongs in your stack.

The constraint only holds where Guidance controls sampling

The two backends that give you the real thing are guidance.models.LlamaCpp, which wraps llama-cpp-python and loads GGUF files, and guidance.models.Transformers, which loads Hugging Face weights. Both expose the logits, so select(), regex-bounded gen() and grammar constraints are enforced token by token. The OpenAI and other remote model classes cannot see logits, so the same calls become suggestions the model may ignore; the catalogue's note that API models limit features is exactly this. Ollama counts as remote here, since Guidance talks to it over HTTP. "Works with local models" means "works with models you load inside the Python process."

The 8 GB RAM figure in the catalogue is the model, not the library. An 8B-parameter instruct model at Q4 quantisation is roughly 5 GB of weights; the library itself is a normal Python install. A 12 GB GPU or a 16 GB CPU-only box is a comfortable floor.

from guidance import models, gen, select

lm = models.LlamaCpp("Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf", n_gpu_layers=-1)

lm += "Classify the support ticket.\nTicket: The invoice was wrong again.\nSentiment: "
lm += select(["positive", "neutral", "negative"], name="sentiment")
lm += "\nOne-line summary: " + gen("summary", regex=r"[^\n]{10,80}", stop="\n")

print(lm["sentiment"], lm["summary"])

The sentiment value will be one of the 3 strings, every time, with no retry loop and no JSON repair. That is the whole pitch in 6 lines.

Token healing is the feature nobody advertises

Prompts that end mid-token (a trailing space, an opening quote, the first half of a URL) push the model into a corner of its vocabulary it never saw during training, and output quality drops in ways that are hard to attribute. Guidance backs up one token at the boundary and lets the model regenerate it, so the join between your text and its text is always a natural tokenisation. You do nothing to enable it, it costs one token per boundary, and on templated prompts with lots of structure it is a measurable quality win for free.

Where it sits next to Ollama and vLLM

Guidance is not a serving layer and does not replace whatever runs your chat UI. It is for pipelines: batch classification, entity extraction, filling tool arguments, anything where a wrong shape breaks the next step. The practical pattern is a worker process that loads its own copy of the GGUF through the LlamaCpp backend while Ollama keeps serving Open WebUI on the same box. The cost is two copies of a model in memory when both are active, which on a single 12 GB GPU means picking a smaller quant for one of them.

If a shared server is the goal, vLLM ships its own guided decoding with regex and JSON schema support, and llama.cpp's server accepts GBNF grammars and JSON schemas natively. For "give me valid JSON matching this schema" you may not need Guidance at all; the structured-output guide walks through those options. Guidance earns its place when Python logic lives between generations: generate a field, branch on it, generate the next, loop while a condition holds. No server-side grammar can express that, because the control flow is your code.

The paradigm is the learning curve

Programs are plain Python built up with +=, and each += returns a new model state rather than mutating the old one, which surprises people who expect a chat history object. The catalogue lists the repository's language as Jupyter Notebook, and that is honest: the project is notebook-first, and most of the 21,714-star community's examples are notebooks. Budget a day to stop fighting it.

The bigger trap is age. The API changed substantially between the early Handlebars-style templating and the current Python-native design, and older blog posts still rank well. If a tutorial shows {{gen 'answer'}} inside a string, it is describing a version you cannot install any more. Pin the version in your requirements file and read the current README rather than anything with a template syntax.

What I'd do

For a one-off "return valid JSON" need against a local model, use llama.cpp's JSON schema support or vLLM's guided decoding and leave Guidance out. Reach for Guidance when the logic between generations is real Python: multi-step extraction, an agent that must choose from an enum before it fills arguments, a form where later fields depend on earlier answers. Run it through the LlamaCpp backend with an 8B Q4 model on a 12 GB GPU, pin the version, and do not bother pointing it at a hosted API, because the only feature that makes it worth learning does not survive the network hop.