vLLM

Last reviewed Aug 25, 2026 · 760 words

vLLM is what you deploy when local AI stops being one person chatting and starts being a service: its continuous batching and PagedAttention memory management let one GPU serve many simultaneous requests at several times the total throughput of Ollama on identical hardware. That's the whole pitch, and it's real — with the equally real corollary that if you're the only user, most of that advantage evaporates and you've traded Ollama's convenience for tuning flags. This guide is about knowing which side of that line you're on, and running vLLM well once you cross it.

What the clever memory tricks actually do

Two mechanisms explain the performance reputation. Continuous batching admits new requests into the GPU's batch as running ones finish, instead of waiting for a whole batch to complete — so ten users' requests interleave efficiently rather than queueing. PagedAttention manages the KV cache (the per-conversation attention memory) in non-contiguous pages, eliminating the fragmentation that wastes VRAM in naive servers and enabling tricks like prefix sharing when many requests share a system prompt. The upshot in numbers: where a llama.cpp-style server might sustain a few hundred tokens/second aggregate under concurrent load, vLLM commonly reaches 2–4× that on the same card — the gap the vLLM vs llama.cpp breakdown measures properly. At concurrency one, the gap shrinks toward noise.

Launching it, and the flags that matter

The Docker route wants the NVIDIA Container Toolkit and generous shared memory:

docker run --gpus all --shm-size=8g -p 8000:8000 \
  -v ./models:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-14B-Instruct-AWQ \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90

Three flags do most of the governing. --max-model-len caps context length — and context is the hidden VRAM cost: vLLM pre-allocates KV-cache space, so a 128k window you never use steals memory that could serve concurrent requests. Set it to what you actually need. --gpu-memory-utilization (default 0.9) is how much VRAM vLLM claims; lower it if the GPU is shared. And the model choice itself: vLLM serves quantized models (AWQ, GPTQ, FP8) but its native habitat is safetensors from Hugging Face, not the GGUF files your Ollama library holds — plan on re-downloading models. The server speaks the OpenAI API on port 8000, so Open WebUI and every other client connect exactly as they did before, which makes the migration invisible above the serving layer.

VRAM budgeting: weights are only the beginning

The VRAM math for single-user inference — model weights plus a little — understates vLLM's appetite, because serving capacity is KV-cache space: every concurrent conversation holds pages proportional to its context. Practical outcome on a 24 GB card: a 14B AWQ model (~9 GB weights) leaves ~12 GB of cache, comfortably serving a small team at 16k contexts; a 32B model on the same card serves fewer simultaneous users than the weights alone suggest. When vLLM logs "no available KV cache blocks" or requests queue, that's the budget speaking — shrink --max-model-len, quantize harder, or accept the concurrency ceiling. Startup is also honest about being production software: model loading and graph compilation take minutes, not Ollama's seconds, which is why it suits the always-on service pattern rather than casual model-hopping.

When vLLM is the wrong answer

Solo chat and model experimentation: stay on Ollama — instant model switching and keep-alive convenience beat throughput you can't use. Odd hardware, CPU-only, or Macs: llama.cpp territory. vLLM earns its complexity at genuine concurrency: the household's self-hosted AI stack once several people use it daily, an internal tool hitting the API in parallel, or batch pipelines where aggregate tokens/second is money. The API has no authentication, same as Ollama — --api-key exists, use it, and keep port 8000 behind the tailnet regardless.

What I'd do

Keep Ollama for experimentation, and stand up vLLM the day the GPU serves more than two humans or any automated pipeline: one pinned model chosen deliberately, AWQ-quantized, --max-model-len sized to real usage, API key set, Open WebUI pointed at it. Run the before/after with the benchmarking method rather than vibes — concurrency is where the 3× lives, and measuring it is the difference between deploying infrastructure and collecting software.