Chroma

Last reviewed Aug 26, 2026 · 776 words

Most people who "self-host Chroma" never run a Chroma server, and that is the correct choice. pip install chromadb gives you a persistent vector store on local disk with a 5-line API, and the client-server mode exists for the day a second process needs the same collection. Under roughly 1 million vectors on one machine it is the least effort you will spend on any part of an AI stack; above that it is the part you replace, and the project is honest that it is not built for huge scale.

Embedded first: the whole database is a directory

There is no daemon, no port, no config file. You name a folder and it becomes the database:

import chromadb
client = chromadb.PersistentClient(path="./chroma-data")
notes = client.get_or_create_collection("notes")
notes.add(ids=["n1", "n2"],
          documents=["Caddy renews certs automatically", "Restic prunes on Sundays"],
          metadatas=[{"topic": "proxy"}, {"topic": "backup"}])
hits = notes.query(query_texts=["how do certificates get renewed"],
                   n_results=3, where={"topic": "proxy"})

The first add downloads a small ONNX copy of all-MiniLM-L6-v2 (about 80 MB, 384 dimensions) and embeds on CPU, so a laptop with no GPU works. Swap in an embedding function backed by Ollama and a model like nomic-embed-text when you want better recall on technical text. The where filter on metadata is the feature people underrate: it lets one collection serve several projects, or restrict retrieval to a date range, without a second index. Backing up is tar on the folder while nothing is writing.

The server is one process on port 8000

When a second service needs the collection, or the app runs in a different container from the data, start the same engine as a server:

chroma run --path /srv/chroma-data --port 8000

or run the chromadb/chroma image with a persistent volume. Client code changes one line, chromadb.HttpClient(host="chroma", port=8000), and the rest of the API is identical, which is the strongest argument for prototyping embedded and promoting later. Two cautions. There is no authentication unless you configure the token-based option through environment variables, so the server belongs on a Docker network or behind Tailscale, never on a public port. And the 1 GB RAM guidance is a floor: the HNSW index lives in memory, and 1 million 768-dimensional float32 vectors are about 3 GB before the index overhead, so the server's RAM scales with the corpus, not with traffic.

Where it already lives in your stack

You may be running Chroma without knowing it. Open WebUI uses it as the default vector store for uploaded documents and its knowledge collections, sitting inside the Open WebUI data directory, which is why that folder grows when you upload a 400-page PDF. AnythingLLM and Dify list it as a supported backend. LangChain and LlamaIndex both ship first-party integrations, so nearly every RAG tutorial you will read uses Chroma for the retrieval half, and the Python-native API is why: add, query, where, done. If you are writing your own RAG service against a local model, Chroma is the store I would reach for first.

The ceiling and who to move to

Chroma is single-node, keeps the index in RAM, and offers no replication. That is fine for a personal knowledge base, a homelab documentation bot, or a product with tens of thousands of documents. It stops being fine when you need filtered search over 10 million vectors, quantisation to fit a big corpus into modest RAM, or a second node for availability. At that point Qdrant is the natural step up: a similar mental model, a real filter engine, scalar and binary quantisation, and a server that was designed as a server from day one. If you already run Postgres, pgvector saves you a service entirely. The vector database comparison works through the choice in more detail; the summary is that hosted Pinecone buys you nothing at homelab scale that Chroma embedded does not give you for free.

What I'd do

Start embedded, in the same process as whatever is doing the retrieval, with the data folder inside the app's volume so it is backed up with everything else. Promote to chroma run behind the Docker network only when a second consumer appears. Set a reminder at 500,000 vectors to measure RAM and query latency, and if either is trending badly, migrate to Qdrant then rather than at 5 million when the migration is painful. For 9 out of 10 self-hosted RAG projects, that reminder never fires.