llama.cpp vs vLLM

Last reviewed Aug 26, 2026 · 644 words

llama.cpp exists so a model can run anywhere; vLLM exists so a model can serve everyone. Both expose an OpenAI-compatible HTTP endpoint, both started in 2023, and they optimize for opposite ends of the same problem — llama.cpp (125,660 stars, MIT, plain C/C++) squeezes quantized models onto modest hardware, while vLLM (90,059 stars, Apache-2.0, Python) squeezes maximum concurrent throughput out of datacenter-class GPUs. The catalogue rates both Hard, but they're hard in unrelated ways: one asks you to compile and script, the other asks you to own and tune expensive silicon.

Hardware decides most matchups before benchmarks do

llama.cpp pioneered the GGUF quantized format and runs it on nearly anything: x86 and ARM CPUs, Apple Silicon via Metal, NVIDIA via CUDA, AMD via ROCm and Vulkan. The 8 GB catalogue minimum is honest — a 4-bit 7-8B model fits there and is usable on CPU alone. vLLM's 16 GB minimum understates its appetite; it is a GPU engine, practically requiring a capable NVIDIA card, and rewards you with PagedAttention memory management and tensor parallelism across multiple cards. So the first question is inventory, not preference: no big GPU means llama.cpp by default, and a multi-GPU server wasted on llama.cpp means you bought vLLM hardware and didn't install it.

Single-stream versus saturation

For one user generating one response, well-quantized llama.cpp on a decent GPU is quick, and on Apple Silicon it's often the only serious option. The gap opens under concurrency: vLLM's continuous batching admits new requests into the running batch instead of queueing them, which is how one card serves dozens of simultaneous sessions at high aggregate tokens per second. llama.cpp's built-in llama-server does offer parallel slots and works fine for a household or small team, but it isn't chasing that ceiling. If you want to measure rather than trust, benchmarking local LLMs covers how to get numbers that survive contact with your own workload.

Day-2 pain, itemized

llama.cpp's cons in the catalogue — command-line focused, frequent breaking changes — match my experience: flags get renamed, binaries get reorganized, and a GGUF that ran last month may want a rebuilt binary this month. It behaves like a fast-moving library, because it is one. vLLM's cons are complex tuning and GPU dependence: memory-utilization fractions, context-length caps, and quantization schemes (AWQ, GPTQ, FP8) that interact with your exact card. Neither ships authentication worth trusting on the open internet; both belong behind a reverse proxy or VPN.

You may be running llama.cpp already

A large slice of the local-AI ecosystem is llama.cpp with a wrapper on top: Ollama builds on it, as do KoboldCpp and LM Studio. If the CLI is the part that puts you off, the llama.cpp vs Ollama comparison covers when the wrapper is the smarter buy — you trade some control and bleeding-edge flags for model management that doesn't break.

Decision table

What I'd do

Match the engine to the metal. On anything short of a 16 GB+ NVIDIA card I'd run llama.cpp — usually via Ollama unless I needed its raw flags — and I'd pin the version I compiled. On a dedicated GPU server with multiple users, I'd take vLLM's tuning session as the entry fee and be done in an afternoon; nothing else self-hosted turns that hardware into as much throughput.