Local LLM Runners

Last reviewed Aug 26, 2026 · 456 words

Every serious LLM runner is free and open-source, and the easy ones install in minutes; the hard part is never the software, it is whether the model fits your machine. The floor across this category is 8 GB of RAM, which in practice means a quantized 7–8B model taking roughly 4–5 GB and leaving room for the OS. GPU VRAM raises the ceiling and the speed, but the arithmetic comes before the shortlist.

How to choose a runner

Two questions settle it. Who consumes the tokens: just you at a desk, or many users and services hitting an API? A single user wants convenience — one command to pull a model, sane defaults. A team wants throughput, which is a different engineering problem: vLLM exists because batching dozens of concurrent requests efficiently (its PagedAttention memory management, continuous batching) is nothing like serving one chat. It wants 16 GB and, practically, a GPU. Second question: how much control you want over inference. Every option here exposes an OpenAI-compatible API, so ignore that as a differentiator — it is table stakes.

Where to start

Ollama (MIT, 179,437 stars) is the default for a reason: ollama run pulls and starts a model in one command, the API on port 11434 plugs into nearly every chat UI, and it runs on CPU or GPU. Two operational notes from the catalogue: models are multi-gigabyte, so put the volume on a disk with room to grow, and the API has no authentication — never publish port 11434 to the internet. llama.cpp is the engine underneath much of the local-AI ecosystem and the pick when you want control: it pioneered the GGUF quantized format, runs on remarkably modest CPUs, and exposes every inference knob Ollama hides — at the cost of a command-line workflow and frequent breaking changes. vLLM is the answer once "runner" means "serving infrastructure for a team".

Gateways are not runners

LiteLLM sits in this category with 57,278 stars and runs no models at all: it is a proxy that fronts 100+ LLM providers behind one OpenAI-compatible endpoint, with spend tracking and key management. Useful — I run one — but installing it expecting local inference is this category's classic mix-up. If a tool's requirements fit in 1 GB of RAM, it is routing tokens, not generating them.

What I'd do

Start with Ollama on whatever hardware you have, even CPU-only; a small model at reading speed teaches you more than a spec sheet. Move to llama.cpp when you find yourself fighting Ollama's defaults, and to vLLM only when more than a handful of people depend on the endpoint.