VRAM needed = weights + KV cache + activations + overhead. In usable form: weights are parameters × bytes-per-weight (0.6GB per billion parameters at Q4_K_M, 2.0 at FP16); KV cache is 2 × layers × KV heads × head dimension × bytes × context length; activations and runtime overhead eat another 1–1.5GB. So an 8B model at Q4 with 8k context is 4.9 + 1.0 + 1.2 ≈ 7GB — it fits an 8GB card with nothing to spare. That's the whole method; the rest of this post is the coefficients and the worked examples.
Weights: bytes per parameter by quant
| Format | GB per 1B params (approx) | 8B model | 32B model | 70B model |
|---|---|---|---|---|
| FP16/BF16 | 2.0 | 16.0GB | 64GB | 140GB |
| Q8_0 | 1.1 | 8.5GB | 34GB | 75GB |
| Q5_K_M | 0.70 | 5.7GB | 23GB | 50GB |
| Q4_K_M | 0.60 | 4.9GB | 19GB | 40GB |
| IQ3_XXS | 0.45 | 3.6GB | 14GB | 31GB |
The coefficients are slightly above the nominal bit width because GGUF keeps embeddings and some sensitive layers at higher precision. What each step costs in quality is its own topic — quantisation explained has the numbers — but as a fitting rule, Q4_K_M is the planning default and anything below Q4 is a compromise you should benchmark before trusting.
KV cache: the part everyone forgets
Per token of context, the cache costs 2 (K and V) × n_layers × n_kv_heads × head_dim × bytes per element. For Llama-3.1-8B — 32 layers, 8 KV heads (grouped-query attention), head dim 128 — that's 2 × 32 × 8 × 128 × 2 bytes = 128KB per token at FP16. Now scale it:
| Context | 8B (128KB/tok) | 70B (320KB/tok, 80 layers) |
|---|---|---|
| 4k | 0.5GB | 1.25GB |
| 8k | 1.0GB | 2.5GB |
| 32k | 4.0GB | 10GB |
| 128k | 16GB | 40GB |
Note the punchline in the first column: an 8B model's KV cache at 128k context (16GB) is more than three times the size of its Q4 weights (4.9GB). "Can I run this model" is the wrong question — it's "can I run this model at this context length". Two mitigations: quantise the cache (q8_0 halves it for approximately zero chat-quality cost, q4_0 quarters it with some risk on long-context recall), and be aware that architecture matters — models using MLA (DeepSeek-style latent attention) compress KV by an order of magnitude, which is why their long-context memory numbers look implausibly good.
Multiply by concurrency: parallel request slots each hold their own context, so a server running 4 slots at 8k on that 8B model budgets 4GB of KV, not 1GB. This is the arithmetic behind the Ollama serving settings.
Overhead: the missing gigabyte
Three things sit outside the formula. CUDA context and runtime buffers: 300–600MB. Inference activations: a few hundred MB at chat-sized batches. And if the GPU also drives your desktop, the compositor and browser take 0.5–1.5GB before you load anything. Budget 1GB headroom on a headless box, 2GB on a desktop. When a model "fits" by weight math but Ollama offloads layers to CPU anyway, this overhead is almost always the reason. Verify with ollama ps — it prints the actual split, and anything less than 100% GPU means you're paying a large speed penalty.
Worked examples
8GB (laptop GPU, RTX 4060): 8B Q4_K_M (4.9) + 8k FP16 KV (1.0) + overhead (1.2) = 7.1GB. Fits headless; on a desktop, drop context to 4k or quantise the cache. A 12–14B does not fit without heavy offload.
12GB (RTX 3060/4070): Qwen-class 14B Q4_K_M (~8.7) + 8k KV at q8_0 (~0.8) + overhead = ~10.7GB. Comfortable. This card is the budget sweet spot in the hardware tier list for exactly this reason.
24GB (3090/4090): 32B Q4_K_M (19) + 8k q8_0 KV (~1.3) + overhead = ~21.5GB — fits, tightly, and 16k context does not. Alternative spend of the same VRAM: an 8B at FP16 with ~48k context for long-document work. Same card, two very different machines.
70B: Q4_K_M weights alone are 40GB. That's 2×24GB with the model split across cards (tensor parallel in vLLM, or llama.cpp row split), a 48GB workstation card, or Apple unified memory. On a single 24GB card with CPU offload expect single-digit tokens per second — possible, not pleasant. Apple Silicon plays by different rules: unified memory serves as VRAM, with macOS allowing the GPU roughly 65–75% of total RAM by default, so a 64GB Mac behaves like a ~45GB card that is also running your browser.
One more consumer of VRAM worth knowing: vLLM preallocates ~90% of the card by default for its paged KV pool. That's by design (gpu_memory_utilization), and it means vLLM "using" 22GB for an 8B model is normal, not a leak.
Do the math before the download
The two-minute version for any model card you're reading: parameters × 0.6GB for Q4 weights; look up layers × KV heads × head dim in the config (or assume 128KB/token per 8B-equivalent as a Llama-family heuristic); multiply by the context you actually intend to run and the requests you'll run in parallel; add 1–1.5GB. If the total clears your VRAM with 10% to spare, pull the model with Ollama and check ollama ps says 100% GPU. If it doesn't clear, buy the smaller quant, shrink the context, or accept the offload penalty knowingly — the arithmetic is cheaper than the disappointment either way.