AnythingLLM

Last reviewed Aug 26, 2026 · 804 words

AnythingLLM collapses the usual five-service RAG stack — chat UI, embedder, vector database, document pipeline, agent framework — into one Docker container that idles around 1 GB of RAM. That consolidation is the entire reason to pick it. You lose the fun of wiring components together and gain a private NotebookLM-style workspace your whole household or team can use by Friday. Mintplex Labs ships it under MIT, it sits at 65,203 GitHub stars, and the important decisions are all made in the first hour: which model backend, which deployment shape, and whether the defaults it bundles are good enough for your corpus.

Batteries included, and the batteries are fine

Out of the box the container embeds documents with a built-in local embedder and stores vectors in LanceDB, which runs inside the app with no extra service. For most personal libraries — a few thousand PDFs, notes, scraped pages — that default stack is genuinely sufficient, and I would not replace it preemptively. The escape hatches exist when you need them: it speaks to external vector stores like Qdrant, Chroma, Milvus, and others, which starts to matter around the point where multiple apps share one vector database or collections grow past what one container should hold. The vector database guide covers when that point actually arrives; it is later than most people think.

The model backend decides everything about quality

AnythingLLM generates nothing itself — it orchestrates whichever LLM you point it at, and the catalogue's "quality depends on models" con deserves promotion to headline. Point it at Ollama on the same box and everything stays local; point it at OpenAI, Anthropic, or another API and you trade some privacy for answer quality on hard questions. The pragmatic split I use: local model for anything sensitive (that is why you are self-hosting), API for workspaces holding public material where you just want the best summaries. You can mix — the provider is set per workspace, not globally — and switching later takes a minute, so start local and upgrade only where the answers disappoint.

Workspaces are the security model, not just the filing system

Each workspace bundles its own documents, chat history, provider settings, and users. Retrieval never crosses workspace boundaries, which means a "household documents" workspace and a "work contracts" workspace can live on one server without bleeding context into each other's answers. This is more useful than it sounds: RAG systems fail embarrassingly when the retriever surfaces the wrong domain's documents, and hard separation is the cheapest fix. Seed each workspace with 10–20 documents, ask questions you know the answers to, and only then bulk-import — citation checking on a small corpus is how you calibrate trust.

Docker or desktop: multi-user lives on one side only

There are two products wearing one name. The desktop app (Windows, macOS, Linux) is single-user, installs like any application, and is the right call for a personal research tool. The Docker deployment (the server listens on port 3001) is where multi-user workspaces, authentication, and the API live — that is the version you put behind a reverse proxy with TLS and share with a team. People regularly start on desktop, accumulate a curated document library, and then discover the server is a separate instance to populate. If there is any chance others will use it, start with Docker on day one and reach it from your laptop over the LAN.

Hardware math: 4 GB is the floor, not the budget

The catalogue's 4 GB minimum covers AnythingLLM itself with embedded LanceDB and a remote or API model. Add a local model via Ollama and the real budget appears: roughly 5 GB extra for a 7B–8B model at 4-bit quantization, 10 GB+ for the 13B class, and embedding large document batches spikes CPU besides. A 16 GB mini PC runs the full local stack comfortably for one or two users; an 8 GB box should either use an API backend or accept small models. Agents — the built-in web-browsing and tool-using flows — add API calls and latency but no meaningful memory, so they don't change the sizing.

What I'd do

Docker deployment on a 16 GB machine, Ollama alongside with an 8B instruct model, defaults otherwise untouched: built-in embedder, LanceDB, one workspace per domain of your life. Proxied with TLS if anyone else joins, LAN-only if not. Spend your first evening asking questions whose answers you can verify, not importing everything you own. If you outgrow it, the growth path is real — external Qdrant, bigger models, the AnythingLLM vs RAGFlow question for heavier document processing — but the honest report from most self-hosters is that the defaults quietly handle years of personal use.