Quantisation Explained: GGUF, AWQ, and What You Actually Lose

Q4_K_M is the right default: roughly 70% smaller than FP16, a quality cost in the low single digits on most benchmarks, and the format Ollama ships when you pull a model without asking for anything else. The two rules worth memorising: below 4 bits, losses become task-dependent gambling; and a bigger model at 4-bit almost always beats a smaller model at 8-bit in the same memory — a 32B at Q4 outperforms a 14B at Q8 on essentially everything. Everything else in this post is detail under those two rules, plus how to check them on your own workload instead of trusting mine.

What quantisation actually does

Weights train at 16-bit precision; quantisation stores them in fewer bits grouped into blocks, each block keeping a scale factor (and sometimes a minimum) to reconstruct approximate values at runtime. Activations stay in higher precision — this is why quality survives at all. You gain twice: memory (the model fits) and speed, because decode is memory-bandwidth bound, so reading a quarter of the bytes per token means roughly 3–4× faster single-stream generation than FP16 on the same card. The K-quants in llama.cpp's GGUF format add per-layer intelligence, spending more bits on sensitive tensors — the _M in Q4_K_M marks that mixed allocation, and it's why Q4_K_M beats the naive 4-bit formats of 2023 by a wide margin. The i-quants (IQ4_XS, IQ3, IQ2) use codebooks calibrated with an importance matrix to stay coherent below 4 bits; treat everything at or under 3 bits as "runs, with caveats you must test".

Format guide: which file to download

The practical split: GGUF for everything self-hosted (its CPU/GPU layer split is unique — a 32B model can run with 30 layers on a 12GB card and the rest on CPU), AWQ or FP8 for production GPU serving. The llama.cpp repository documents the GGUF variants; ollama tags map onto them directly, so ollama pull qwen3:14b-q8_0 gets you Q8 where the default tag is Q4_K_M.

Size and quality by bit depth

For an 8B model (weights only — KV cache is on top, and VRAM math covers that side):

Two footnotes that move decisions. Small models degrade faster — a 3B at Q4 loses visibly more than a 70B at Q4, so the smaller the model, the higher I keep the bits. And the table's "quality cost" is an average that hides variance, which is the entire reason for the next section.

What degrades first (and why vibes-testing lies)

Quantisation error is not uniform across capabilities. First to go: maths and multi-step arithmetic, code generation edge cases (off-by-one logic, rare API details), long-context recall, instruction precision on fiddly format requirements, and low-resource languages. Last to go: chat fluency, tone, general knowledge phrasing. A Q3 model chats beautifully while quietly becoming worse at the thing you actually deployed it for — which is why "I tried it and it seems fine" is the least trustworthy sentence in local AI.

How to A/B a quant honestly

Compare on your task, deterministically, at sample sizes that mean something. Temperature 0, fixed seed, 200+ prompts drawn from your real workload, exact metrics (pass@1 for code, exact-match for extraction, rubric scores for prose) — the same discipline as benchmarking local models generally, run once per candidate quant. Anything under ~100 prompts and you're reading noise; a 2% real regression needs hundreds of samples to see.

For a task-independent check, llama.cpp measures KL divergence of the quant's token distribution against the FP16 model — closer to zero means more faithful:

# 1. record baseline logits from the full-precision model
./llama-perplexity -m model-f16.gguf -f wiki.test.raw \
  --kl-divergence-base f16-logits.bin
# 2. compare a candidate quant against them
./llama-perplexity -m model-q4_k_m.gguf \
  --kl-divergence-base f16-logits.bin --kl-divergence

KL divergence and perplexity deltas rank quants reliably but don't tell you whether your task survives — I use KL to shortlist (typically Q4_K_M vs Q5_K_M vs IQ4_XS) and the task eval to decide. Budget an evening; it's one-time per model and it replaces every forum argument about whether Q4 is "basically lossless" (answer: usually, for you, maybe — measure).

Bottom line

Default Q4_K_M. Go Q6_K or Q8_0 when VRAM is spare and the task is code or maths. Go IQ4_XS or Q3_K_M only to fit a decisively bigger model, and eval before trusting anything at 3 bits or below. Prefer the largest model that fits at Q4 over a smaller one at higher precision, and spend one evening on a temperature-0, 200-prompt A/B before shipping any quant to a task that matters — the failure mode of quantisation isn't a model that breaks, it's a model that's 4% worse at exactly what you needed while chatting like nothing happened.