LocalGPT

Last reviewed Aug 26, 2026 · 918 words

Open WebUI running on top of Ollama does about 90% of what LocalGPT does, and adds a login page, multiple users, and a document upload button. The 10% LocalGPT keeps is the reason it has 22,207 stars: every stage of the retrieval pipeline is a readable Python file you can change, from which embedding model chunks your PDFs to how many passages get handed to the language model. It is a workbench for learning and tuning retrieval-augmented generation, not a service you put behind a reverse proxy. Judged as a workbench it is excellent. Judged as an app, it loses, and the catalogue's "single-user" tag is the tell.

What you're installing: a folder, an ingest script, a chat script

There's no image to pull in the usual sense, although a Dockerfile exists. The typical setup is a checkout and a virtual environment:

git clone https://github.com/PromtEngineer/localGPT.git && cd localGPT
pip install -r requirements.txt
cp ~/papers/*.pdf SOURCE_DOCUMENTS/
python ingest.py --device_type cuda        # cpu, or mps on Apple silicon
python run_localGPT.py --device_type cuda

ingest.py walks SOURCE_DOCUMENTS/, splits each file into chunks, embeds them, and writes a local vector store to disk. run_localGPT.py loads a model and starts a terminal question loop against that store. There is a Streamlit UI and a small API server in the repo as well, both thin wrappers over the same two steps. The project has been reorganised more than once since 2023, so check the README's current entry points before copying commands from a blog post, including this one. Nothing leaves the machine at any stage, which is the property the whole project is built around and the one to verify if you fork it: watching the network interface during a session is the honest test.

8 GB is the floor; the GPU is the difference

The catalogue's 8 GB minimum is for CPU-only inference of a small quantised model with a small document set, and at that size a question takes 30 seconds to a minute to answer. With an 8 GB or larger GPU the same question takes a few seconds, and ingestion of a few hundred pages drops from tens of minutes to a couple. Apple silicon sits in between via the mps device. The embedding model also lives in memory during ingest, so the practical planning number for a comfortable experience is 16 GB of system RAM plus whatever the GPU brings. The hardware guide for self-hosted LLMs has the model-size-to-VRAM table; LocalGPT doesn't change that maths, it just makes you feel it more directly.

Ingest once, ask many: what the pipeline does

Retrieval-augmented generation is three steps and LocalGPT exposes all three. Chunk: documents are split into overlapping windows of a few hundred tokens. Embed and store: each chunk becomes a vector via a local embedding model and goes into a Chroma store in a DB/ folder. Retrieve and answer: your question is embedded the same way, the top few chunks by similarity are pulled, and they're pasted into the prompt ahead of your question. Every one of those numbers (chunk size, overlap, how many chunks, which embedding model, which LLM) is a constant near the top of a file. That is the value: change one, re-ingest, and see the answers change. If you've wondered whether retrieval or fine-tuning is the right tool for your documents, the RAG vs fine-tuning post is the framing, and LocalGPT is the cheapest place to test it.

Where it loses to Open WebUI and AnythingLLM

There's no authentication, so it can't safely be exposed. There's one user and one document store. Adding a document means re-running ingest, and the store isn't incremental in any pleasant way. There's no conversation history worth the name, no citations with page numbers in the terminal loop, and the Streamlit UI is a demo rather than a product. Open WebUI gives you document chat with users, RAG settings in a settings page, and a proper interface; AnythingLLM is the same idea with workspaces per project and an agent layer. Both talk to Ollama, and both are what I'd install for a household.

Where it still wins

Air-gapped machines: LocalGPT has no external calls once the models are downloaded, and the whole thing fits on a USB stick. Understanding: an afternoon reading its source teaches more about retrieval than a month of using a polished UI. Component swapping: trying a different embedding model or splitter is a one-line change, which in the bigger apps is a feature request. And research on a single, sensitive document set where you want to know exactly what happens to every byte.

What I'd do

For serving a family or a team, Ollama plus Open WebUI, and never look back. For myself, when I want to know why the answers to a specific set of manuals are bad, a LocalGPT checkout on the machine with the GPU, an hour spent tuning chunk size and retrieval count, and then I'd carry the findings back into the settings page of the app people actually use. That is the honest role for it: a tuning bench, not a home for your documents. As a bench it is the best free one I know.