The gap between two 8B models on a public leaderboard is usually smaller than the error introduced by your quant choice, your chat template, and benchmark contamination combined. Leaderboard scores are measured on FP16 weights with a harness you're not using, on test sets the models may have memorised — while you'll run a Q4 file through Ollama with different sampling defaults. The only benchmark that predicts your workload is one built from your workload: 100 items, one evening, instructions below.

Three ways public scores lie to you

Contamination. Popular benchmarks leak into training data. The cleanest demonstration remains GSM1k (Scale AI, 2024), which rebuilt GSM8k with fresh problems of matched difficulty and found some models dropped up to 13 points — they'd partially memorised the original. Assume any headline number on MMLU, HumanEval, or GSM8k is an upper bound, tightest for the benchmarks oldest and most copied. A crude sniff test you can run locally: paste the first half of a benchmark question and ask the model to continue. Verbatim completion of the question text is memorisation, observed directly.

Your quant isn't their quant. Leaderboards score FP16; you'll run Q4_K_M. The degradation is real and unevenly distributed: chat and summarisation lose almost nothing at Q4, while code generation and multi-step arithmetic can drop 5–15% (estimates; it varies by model and task, which is exactly the point). Smaller models degrade more per bit. Benchmark the artifact you deploy — the specific GGUF file at the specific quant — and when comparing two models, compare them at the same quant class. The mechanics of what each bit depth costs are in quantisation explained.

Template and settings bugs. A wrong or subtly outdated chat template silently costs several points — the model sees malformed special tokens and degrades without erroring. Sampling defaults differ per runtime: Ollama ships temperature 0.8, which is a fine chat default and a terrible eval default. And Ollama's 4,096-token default context will silently truncate long eval prompts, producing garbage scores that look like model failure. Set temperature 0, set num_ctx explicitly, and diff the template against the model card before believing any number.

Measure speed like you mean it

Speed claims need the prefill/decode split, because the two scale differently and matter to different workloads:

ollama run llama3.1:8b-instruct-q4_K_M --verbose "Summarise: <2000 words>"
# ...
# prompt eval rate:     412.30 tokens/s   <- prefill (long-context reading)
# eval rate:             54.71 tokens/s   <- decode (generation speed)

Rules for numbers worth writing down: run each measurement three times and discard the first (it includes model load); use a prompt length matching your real usage, because decode slows as context fills and an empty-prompt tok/s flatters the result; and on laptops, watch for thermal throttling — a 4090 desktop gives you the same number at 9am and 5pm, an M-series MacBook on a couch does not. Report "54 tok/s decode at 2k context, q4_K_M, Ollama 0.9" or the number is decoration.

The harness you can build tonight

Pull 50–200 real examples from your actual use case — support replies you've written, functions you've asked for, documents you've summarised. Half should be routine, a quarter hard, a quarter edge cases that have burned you. Then wire them into promptfoo, which speaks to Ollama directly:

# promptfooconfig.yaml
providers:
  - ollama:chat:llama3.1:8b-instruct-q4_K_M
  - ollama:chat:qwen3:8b-q4_K_M
defaultTest:
  options:
    temperature: 0
tests:
  - vars: { input: "Refund request, order shipped 3 weeks ago..." }
    assert:
      - type: contains
        value: "30-day"
      - type: llm-rubric
        value: "Polite, cites the returns policy, under 120 words"

Grade mechanically wherever possible — exact match, contains, schema validation, code that runs — and use an LLM rubric judge only where you must, calibrating it against 20 hand-graded examples first (judge mechanics in evals before vibes). Score per task category, never one blended number: a model that wins summarisation by 10 points and loses extraction by 15 is a routing decision, not a ranking.

Interpreting results without self-deception

Fifty items give you roughly ±10% noise; treat differences inside that as ties, and only trust small gaps at 200+ items. Temperature 0 isn't perfectly deterministic on GPUs either — batching and floating-point ordering introduce occasional run-to-run variation — which is one more reason a 2-point gap on 50 items means nothing. Compare quants of the same model before comparing models — if Q4 versus Q6 moves your metric more than swapping models does, your bottleneck is quant choice, and the fix is VRAM budgeting rather than model shopping. Beware of tuning your prompt on the eval set: after a few dozen prompt iterations against the same 100 items you've overfit to them, so hold out 30% you look at only for final comparisons. And re-run the harness when anything changes — runtime version, quant file, template — not just when the model name does; two of my three biggest local regressions came from runtime updates, not models.

What I'd do

Tonight: 50 real examples into promptfoo, temperature 0, num_ctx set, three candidate models at the same quant, plus the --verbose speed runs at your real context length. That's enough to pick a model for your workload with more validity than any leaderboard, because it measures the artifact, the runtime, and the task you'll actually use. Grow the set to 200 as failures accumulate, keep it in git next to your prompts, and check the current open-weight field notes only for the shortlist — never for the decision.