PrivateGPT

Last reviewed Aug 26, 2026 · 999 words

PrivateGPT is a reference implementation of offline document question-answering, not a finished product, and reading it that way saves you an evening. Under the hood it is a Python service built on LlamaIndex that ingests your files into a local vector store (Qdrant by default), runs a model through Ollama or llama.cpp, and exposes an OpenAI-compatible API plus a Gradio web page. The privacy claim is real: nothing leaves the host, no telemetry, no cloud fallback. The setup is as fiddly as the catalogue warns, and if the whole goal is "chat with my PDFs" on a home box, Open WebUI on top of Ollama gives you 80% of it in a tenth of the time. PrivateGPT earns its place when you want that pipeline as an API other software can call.

One install path works reliably: the Ollama profile

The project supports several LLM, embedding, and vector-store backends, selected at install time through Poetry extras and at run time through settings profiles. Picking the wrong combination is where most of the "setup is fiddly" reports come from. The combination I would use on a self-hosted box is Ollama for both the model and embeddings, Qdrant embedded on disk, and the UI:

git clone https://github.com/zylon-ai/private-gpt.git
cd private-gpt
poetry install --extras "ui llms-ollama embeddings-ollama vector-stores-qdrant"
ollama pull llama3.1:8b
ollama pull nomic-embed-text
PGPT_PROFILES=ollama make run

It needs Python 3.11 and Poetry, and a running Ollama on localhost:11434. The ollama profile reads settings-ollama.yaml, where ollama.llm_model and ollama.embedding_model name the models to use; change those two lines rather than editing the base settings. The UI comes up on port 8001 and the API on the same port under /v1/. There is also a Docker Compose file with ollama-cpu, ollama-cuda, and ollama-api profiles if you would rather not manage a Python environment, and it is the path I would take on a machine that already runs Ollama in a container.

Three modes, and only one of them is RAG

The web page offers Query Files, Search Files, and LLM Chat. Query Files is the retrieval-augmented mode: your question is embedded, the closest chunks are pulled from Qdrant, and the model answers with those chunks in context, citing the source file. Search Files skips the model and returns the matching chunks, which is the fastest way to check whether a document was actually ingested and split sensibly. LLM Chat ignores your documents entirely and is just the model. Most disappointment with local RAG comes from people asking Query Files a question whose answer was never in the ingested set, then blaming the model; run the same question through Search Files first and you will know within a second whether the retrieval half found anything.

Ingestion is the slow part and the part worth tuning

Drag-and-drop in the UI is fine for 5 files. For a folder, use the bulk script:

make ingest ./documents -- --watch

That walks the directory, parses PDF, DOCX, Markdown, text, CSV, EPUB, and HTML through LlamaIndex readers, chunks the text, embeds each chunk, and writes to Qdrant. --watch keeps it running and picks up new files. Embedding runs on the same hardware as the model, so ingesting 2,000 pages on CPU is a coffee-and-lunch job while a mid-range GPU does it in minutes. The embedding.ingest_mode setting (simple, batch, parallel, pipeline) trades memory for throughput; parallel with a worker count matching your cores is the usual right answer on a desktop.

Two realities the docs underplay. Scanned PDFs contain no text and ingest as nothing; OCR them first (Paperless-ngx does this as a side effect if you already run it). And retrieval quality depends on chunking more than on model choice, so if answers keep missing context that you know is in the file, adjust chunk size before you download a bigger model.

8 GB is the floor, 12 GB of VRAM is the experience

The catalogue's 8 GB minimum assumes a 7B or 8B model at 4-bit quantisation running on CPU, which produces answers at a few tokens a second on a modern desktop processor: usable, not pleasant. A GPU with 12 GB of VRAM holds the same model plus the embedder with room to spare and answers at conversational speed. Apple Silicon with 16 GB of unified memory sits in between and is the quietest option. The hardware guide for local LLMs has the price-per-token-per-second table if you are buying for this.

The honest state of the project

PrivateGPT was built by Zylon, whose focus moved to a commercial product, and the open-source repository's release cadence slowed sharply after 2024. The code works and the 57,000 stars reflect how many people first met local RAG here, but check the commit history before betting a workflow on it, and pin the Ollama version alongside it. The actively maintained alternatives have absorbed the same idea: Open WebUI has document chat built in, and AnythingLLM packages ingestion, workspaces, and an API with a friendlier setup. The split is laid out in AnythingLLM vs PrivateGPT.

What I'd do

If I needed a local, OpenAI-compatible RAG endpoint for other software to call, PrivateGPT with the Ollama profile, Qdrant on disk, parallel ingest, and a 12 GB card. Ingest with the script, verify every batch with Search Files, and OCR anything scanned first. If I just wanted to ask questions of my own documents from a browser, I would skip it and put Open WebUI on Ollama, which is what I run at home. Either way the model stays on your hardware, which is the promise the name makes and the one part of the project that has aged perfectly.