Haystack

Last reviewed Aug 26, 2026 · 805 words

Haystack does not run; you run things built with it. That distinction stops a lot of confusion, because it appears in self-hosting lists next to chat UIs and inference servers as though docker compose up would produce something to click on. It is a Python framework (pip install haystack-ai) from deepset for wiring retrievers, document stores, and language models into pipelines you then serve yourself. Think of it as the layer between your documents and Ollama, the one that decides how text is chunked, embedded, retrieved, and handed to the model.

What a self-hosted Haystack deployment actually looks like

A working stack has four parts. A document store holds text and embeddings: the in-memory store for experiments, Qdrant, Weaviate, Chroma, OpenSearch, or Postgres with pgvector for anything persistent. An embedding model turns chunks into vectors, which can run locally through Ollama or sentence-transformers on CPU. A generator answers questions, again Ollama for local weights or any OpenAI-compatible endpoint. And your own Python process runs the pipeline, usually behind FastAPI or the project's Hayhooks server, which exposes a pipeline as a REST endpoint with very little code. The catalogue's 2 GB minimum is for the Python side alone; the models and vector database are on top of that and are where the real memory goes.

Pipelines are graphs, and that is the point

The 2.x rewrite made every step a component with typed inputs and outputs, and a pipeline a directed graph connecting them. Indexing is one graph (converter, cleaner, splitter, embedder, writer); querying is another (embedder, retriever, prompt builder, generator). The pieces are swappable without rewriting the rest, which is the honest reason to prefer it over hand-rolled scripts: swapping BM25 for dense retrieval, or one document store for another, is a one-component change. Pipelines also serialise to YAML, so the same definition runs in a notebook, in a container, and in CI. The learning cost the catalogue mentions is real; the concepts of connections and sockets take an afternoon before they feel natural.

Where it beats LangChain, and where it does not

Haystack is smaller, slower-moving, and more opinionated than LangChain, which for a production service is mostly a compliment. The abstractions are fewer and better documented, and deepset has kept a stable 2.x API rather than reshuffling modules every quarter. It is weaker if you want the widest possible catalogue of third-party integrations on day one, or if your application is agent-first with long tool-calling loops rather than retrieval-first; it has agents, but retrieval is what it was built around. If you have not settled whether you need retrieval at all, the RAG versus fine-tuning piece is the decision to make before choosing a framework.

A minimal local pipeline to prove the stack

Enough to check that Ollama, an embedding model, and Haystack talk to each other, with the in-memory store so nothing else needs to be running:

from haystack import Pipeline, Document
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.builders import PromptBuilder
from haystack_integrations.components.generators.ollama import OllamaGenerator

store = InMemoryDocumentStore()
store.write_documents([Document(content="The backup job runs at 02:00 and keeps 14 days.")])

template = "Answer using only this context:\n{% for d in documents %}{{ d.content }}\n{% endfor %}\nQuestion: {{ query }}"
pipe = Pipeline()
pipe.add_component("retriever", InMemoryBM25Retriever(document_store=store))
pipe.add_component("prompt", PromptBuilder(template=template))
pipe.add_component("llm", OllamaGenerator(model="llama3.1", url="http://localhost:11434"))
pipe.connect("retriever", "prompt.documents")
pipe.connect("prompt", "llm")
print(pipe.run({"retriever": {"query": "How long are backups kept?"}, "prompt": {"query": "How long are backups kept?"}}))

That needs pip install ollama-haystack alongside the core package. Replace the retriever with a Qdrant one and the embedder that goes with it when you move past a few thousand documents.

What it is not

It is not a chat interface; pair it with a front end or use Open WebUI if what you want is document chat without writing Python. It is not an inference server, and it is not observability; you will still want traces from something like Langfuse once real users hit it. Haystack's job is narrow and it does that job well.

What I'd do

For a personal document assistant, skip it and run a packaged RAG UI. For anything you will maintain, build on Haystack 2.x with Qdrant as the store and Ollama for both embeddings and generation, serialise the pipeline to YAML, serve it with Hayhooks in a container, and keep the in-memory store for tests only. It is the framework I would bet a production retrieval service on, precisely because it does less than its rivals.