LLMWare

Last reviewed Aug 26, 2026 · 767 words

LLMWare is a Python library you pip install, not a container you point a browser at. If you want a chat window over your documents this afternoon, close this tab and run AnythingLLM or Open WebUI instead. LLMWare earns its 14,851 stars with a narrower promise: a complete retrieval-augmented generation pipeline (parsing, chunking, embedding, retrieval, prompting, fact-checking) tuned for 1B to 7B models that run on a CPU. For a self-hoster who writes Python and wants RAG behaviour they can read and test, that is the right tool. For everyone else it is a detour.

The whole pitch is small models on ordinary hardware

Most RAG frameworks assume you will call a frontier model over an API. LLMWare assumes the opposite: the model is small, local, and probably running on the same box as the documents. The project publishes its own families of fine-tuned models on Hugging Face for exactly this. The BLING series are 1B to 3B instruction models built to answer from a passage without hallucinating beyond it; DRAGON models are 6B to 7B versions of the same idea; the SLIM series are tiny function-calling models that emit structured output (classification, sentiment, named entities) rather than prose. Most ship as GGUF quantisations, which is why the stated 4 GB RAM floor is honest rather than optimistic. On a 4-core mini PC with 16 GB a BLING model answers a retrieved passage in a few seconds; a DRAGON model takes longer but stays usable. No GPU is required, and the hardware guide covers what a GPU would buy you if you add one later.

What a minimal pipeline looks like

The library's four working objects are Library, Query, Prompt, and ModelCatalog. A first pipeline is short:

from llmware.library import Library
from llmware.retrieval import Query
from llmware.prompts import Prompt

lib = Library().create_new_library("contracts")
lib.add_files("/data/contracts")
lib.install_new_embedding(embedding_model_name="mini-lm-sbert", vector_db="chromadb")

hits = Query(lib).semantic_query("termination notice period", result_count=5)
p = Prompt().load_model("bling-phi-3-gguf")
p.add_source_query_results(hits)
print(p.prompt_with_source("What notice period applies?"))

add_files handles PDF, DOCX, PPTX, XLSX, HTML, plain text, and images through its own parser, which is the part that saves the most time. By default the parsed text lands in SQLite and vectors go to an embedded store, so a laptop needs nothing else installed. When the corpus outgrows that you swap vector_db for Qdrant, Milvus, or Postgres with pgvector, and the collection store for Mongo or Postgres, without changing the rest of the code.

It talks to Ollama, and that is usually how I'd run it

You are not locked into LLMWare's own models. ModelCatalog can register any GGUF file, and it has an integration for Ollama so the library becomes the retrieval and prompting layer while Ollama stays your model server. That split is the one I'd use: Ollama already handles model downloads, GPU offload, and concurrency; LLMWare adds the document parsing, the source-grounded prompt construction, and the post-hoc checks (it can compare an answer against the passages it was given and flag numbers that do not appear in the source). Those checks are the feature that most turnkey RAG apps lack and the reason to tolerate writing code.

The honest cons

It is a library, so there is no user interface, no accounts, no scheduler. You wrap it in FastAPI or a cron job yourself. The community is smaller than LangChain's or LlamaIndex's, and the documentation leans on a large folder of example scripts rather than reference pages; expect to read source. Its abstractions are opinionated around its own model families, and while other models work, the fact-checking helpers are tuned for the BLING and DRAGON output style. Apache-2.0 licensing means none of this comes with strings.

What I'd do

Run Ollama on the box with the documents, pip install llmware in a virtualenv beside it, and build the first pipeline against a folder of 50 PDFs with SQLite and the embedded vector store. Measure answer quality with the source-check helpers before touching any bigger model; small models plus good retrieval beat big models plus sloppy retrieval more often than people expect, and the RAG versus fine-tuning write-up explains why. If after a week you find you only wanted a chat box, move to AnythingLLM and keep the parsed corpus. If you wanted a pipeline you can unit-test, stay.