llama.cpp

Last reviewed Aug 26, 2026 · 826 words

Most people who think they want llama.cpp actually want Ollama, and it's worth saying so in the first breath: Ollama wraps this very engine in model management and a one-line install, and for "run a local model with minimum fuss" it is the correct answer. llama.cpp itself — the 125,000-star C/C++ project that created the GGUF format and quietly powers a large slice of the local-AI ecosystem — is what you reach for when you want the layer underneath: exact control over quantization, GPU offload, sampling, and server flags, with performance left on the table by every wrapper above it. The catalogue rates it Hard, and that rating is about assembly, not operation.

GGUF and quantization are the reason it runs on your hardware

llama.cpp's foundational trick is aggressive, clever quantization. A model's weights get compressed from 16-bit floats down to 4-5 bits with surprisingly small quality loss, and the GGUF file format packages the result as a single portable file that memory-maps efficiently. Concretely: an 8-billion-parameter model at the popular Q4_K_M quantization is about a 5 GB file and runs acceptably on a CPU with 8 GB of RAM — the catalogue minimum — while a 70B model at the same quantization needs roughly 40-45 GB and rewards a big-memory machine or a multi-GPU rig. The quantization menu (Q4_K_M as the default sweet spot, Q5/Q6 for quality headroom, and newer low-bit formats for squeezing) is the main decision you'll make; quantization explained unpacks the trade-offs and turns them into a shopping list.

llama-server is a drop-in OpenAI endpoint

The piece self-hosters care most about is llama-server: a built-in HTTP server, listening on port 8080 by default, exposing an OpenAI-compatible chat completions API plus its own native endpoints and a basic built-in web UI. Point any OpenAI-client application at it and most work unchanged. A representative launch line:

./llama-server -m models/llama-3.1-8b-instruct-Q4_K_M.gguf -ngl 99 -c 8192 --host 0.0.0.0

-ngl 99 offloads all layers to the GPU (set it lower to split between GPU and CPU when VRAM runs short — partial offload is a llama.cpp specialty), and -c sets context length, which costs memory linearly, so don't reflexively max it. The server does concurrent requests with continuous batching these days, though for many-user production serving on big GPUs, vLLM remains the stronger tool — that boundary is drawn in llama.cpp vs vLLM.

The hard part is the build, and you can skip it

The Hard rating is earned at compile time. Getting the right backend — CUDA for NVIDIA, Metal on Apple Silicon (where llama.cpp is exceptional), ROCm for AMD, Vulkan as the portable fallback — means a CMake build with the right flags and the right toolkit versions installed, and CUDA version mismatches are the classic time sink. Three legitimate shortcuts: prebuilt release binaries from the project's GitHub releases, the official Docker images with CUDA variants, and package managers (Homebrew on macOS is a one-liner). Models come separately — Hugging Face hosts GGUF conversions of essentially every open model, so you download a file, not a repository. If you compile, -DGGML_CUDA=ON and 20 minutes gets you a binary tuned to your exact machine, which is rather the point of the exercise.

It moves fast and breaks flags

The project ships commits daily and versions itself by build number rather than stately releases. The payoff is that new model architectures often work within days of release; the cost is that flags get renamed, defaults change, and the native API surface shifts — the standard warning about frequent breaking changes is accurate. Two habits keep this painless: pin a known-good build (a git tag or Docker image digest) for anything other services depend on, and reread --help after every upgrade instead of trusting old scripts. The OpenAI-compatible endpoint is the most stable surface it has, so integrate against that rather than the native API where you can.

What I'd do

Decision rule: for one user who wants minimum fuss, Ollama; for many concurrent users on datacenter GPUs, vLLM; for everything in between, plus any hardware the big frameworks ignore, llama.cpp — the fuller version of that split is in llama.cpp vs Ollama. My own setup is the prebuilt binary or Docker image, an 8B-class model at Q4_K_M as the daily driver, llama-server pinned to a known build and fronted by my reverse proxy, upgraded deliberately about once a month. Learn it even if you deploy a wrapper: every layer above inherits its behavior, and when something misbehaves at 2 a.m., this is the layer where the answer lives.