vLLM vs llama.cpp: Pick by Workload, Not by Hype

If you serve many concurrent requests on a GPU that holds the whole model, run vLLM. If you run one or two streams on whatever hardware you actually own — a Mac, a 12GB card, a CPU box — run llama.cpp. The crossover sits around 4–8 sustained concurrent requests: below that the two are within 10–15% of each other and llama.cpp's operational simplicity wins; above it, vLLM's continuous batching pulls away fast, reaching 5–10× the aggregate throughput at 32 concurrent streams. Everything else is detail, but the detail decides edge cases, so here it is.

Two designs for two problems

llama.cpp is built to run models where they otherwise wouldn't fit: GGUF quantisation down to 2–4 bits, partial GPU offload (put 30 of 80 layers on an 8GB card, the rest on CPU), Metal on Apple Silicon, Vulkan, plain AVX2. It ships as a single self-contained binary, and llama-server exposes an OpenAI-compatible API. Its batching is slot-based and modest.

vLLM is built to saturate accelerators: PagedAttention manages KV cache like virtual memory so hundreds of requests share a GPU without fragmentation, and continuous batching admits new requests mid-generation instead of waiting for a batch boundary. The price: the model must fit in GPU memory (or across GPUs with tensor parallelism), CUDA/ROCm is effectively required, and it's a Python service with a real dependency tree. It also preallocates ~90% of VRAM by default — that's the gpu_memory_utilization setting, not a leak.

The comparison that matters

Rough numbers from an RTX 4090 running a Llama-3.1-8B-class model (estimates from my bench runs; rerun the method below on yours): single stream, llama.cpp Q4_K_M decodes ~120 tokens/s versus vLLM AWQ ~110. At 32 concurrent requests, vLLM sustains 2,000–2,800 tokens/s aggregate with p95 time-to-first-token under a second; llama.cpp with 8 slots manages 400–500 tokens/s and TTFT grows with the queue.

Benchmark methodology you can rerun

Most vLLM-vs-llama.cpp numbers on the internet are broken in one of three ways: different quants (FP16 vs Q4 is not a runtime comparison), prefill counted in one and not the other, or unlimited client concurrency that measures the queue, not the engine. The fair fight:

# llama.cpp: raw engine speed, prefill and decode reported separately
llama-bench -m llama-3.1-8b-instruct-q4_k_m.gguf -p 1024 -n 256

# llama.cpp serving: 8 slots, 16k total context
llama-server -m llama-3.1-8b-instruct-q4_k_m.gguf -c 16384 -np 8

# vLLM serving
vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 16384

# load test both through the same harness
vllm bench serve --backend openai-chat --base-url http://localhost:8000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --num-prompts 200 --request-rate 8

Hold constant: quant class (Q4_K_M vs AWQ-INT4 is close enough; Q4 vs FP16 is not), context length, sampling settings, and prompt/output length distribution taken from your real traffic. Report TTFT p50 and p95 plus aggregate tokens/s at your actual concurrency — a single blended number hides exactly the trade-off you're choosing between. Watch VRAM during the run too (nvidia-smi dmon): a config that fits at one stream can OOM at 32 when per-request KV allocations stack up, and you want to find that ceiling on the bench, not in production.

Where each one is simply disqualified

llama.cpp is disqualified when you need throughput SLOs under real concurrency, tensor parallelism across 4–8 GPUs, or high-volume guided JSON output — vLLM territory. vLLM is disqualified on Apple Silicon, on CPU-only boxes, and whenever the model doesn't fit VRAM: a 70B Q4 GGUF running split across a 24GB card and system RAM is slow (maybe 4–8 tokens/s) but possible with llama.cpp and simply not a thing vLLM does. Quantisation choice interacts with all of this — see what you actually lose at each bit depth and the VRAM arithmetic before assuming a model fits.

Worth saying plainly: Ollama is llama.cpp with a model manager and lifecycle layer on top, so everything in the llama.cpp column applies to it, and tuning Ollama for serving is really tuning llama.cpp's slots. TGI and SGLang compete in vLLM's bracket; SGLang is worth a bench run if your workload leans on shared prefixes — agent trees and few-shot-heavy templates — where its radix-cache design regularly wins by 20%+.

Pick by concurrent users

Zero to four sustained concurrent streams: llama.cpp (or Ollama on top of it), because the throughput you'd gain from vLLM is throughput you'd never use, and you keep CPU offload, Mac support, and single-binary ops. Eight or more: vLLM, and buy enough VRAM to hold the model — the aggregate tokens/s per dollar is unbeatable once batching is actually exercised. In the 4–8 gap, decide on operational grounds: a team that runs Python services in containers all day picks vLLM; a homelab that wants one process and no pip picks llama.cpp. Benchmarks from anyone else's workload, including mine, are only a starting point — the method above takes about an hour to run against your own traffic shape.