VRAM decides what you can run; everything else decides how pleasant it is. The tiers in early 2026: 8GB runs 7–8B models at Q4, 12–16GB runs the 14B class comfortably, 24GB runs 32B models — and a used RTX 3090 at ~$700 makes this the best value tier in the hobby — 48GB runs 70B at Q4, and Apple unified memory is the cheapest way to fit very large models while accepting slower speeds. CPU-only is genuinely usable for small models and overnight batch work, and genuinely miserable for interactive chat with anything larger than 8B.
The tier table
Prices are used-market estimates as of early 2026; decode speeds are real-world single-stream figures, not theoretical ceilings.
| VRAM | Example hardware | Est. cost | Comfortable ceiling | Decode speed |
|---|---|---|---|---|
| 8GB | RTX 3060 Ti/4060, older laptops | $150–300 | 7–8B Q4 | 35–50 tok/s |
| 12GB | RTX 3060 12GB | $180–250 | 14B Q4 (tight) | 20–30 tok/s |
| 16GB | RTX 4060 Ti 16GB, RX 7800 XT | $350–450 | 14B Q6, 24B Q4 tight | 25–40 tok/s |
| 24GB | RTX 3090 used, 4090 | $650–750 / $1,500+ | 32B Q4 | 25–35 tok/s |
| 48GB | 2× RTX 3090 | ~$1,400 + board | 70B Q4 | 12–18 tok/s |
| 64–128GB unified | Mac mini M4 Pro 64GB, Studio M4 Max 128GB | $2,000–3,500 | 70B Q4–Q8, large MoE | 8–14 tok/s (70B) |
| 80GB+ | A100/H100, 4× 3090 | $8,000+ | 70B Q8, 100B+ | Serving territory |
Model-size arithmetic behind the ceilings: weights at Q4_K_M cost about 0.6GB per billion parameters, and you need headroom for KV cache and the runtime on top — the VRAM math post does the exact accounting, and the quantisation guide covers what Q4 costs in quality (short version: little, and a bigger model at Q4 beats a smaller one at Q8 in the same memory).
The used market: picks and traps
RTX 3090, ~$700. Still the community's default answer five years after launch, because nothing newer offers 24GB anywhere near the price and 936GB/s of bandwidth is what decode actually eats. Buying used: run a 30-minute sustained load test and watch VRAM temperatures — the GDDR6X thermal pads are the known weak point and a $30 repad fixes it. Mining history matters less than the forums fear; undervolt to ~270W and you lose about 5% performance while dropping 80W of heat and noise.
Tesla P40, ~$300. The budget trap that occasionally makes sense: 24GB for half the money, but Pascal-era compute (poor FP16, no flash attention) makes it roughly 3× slower than a 3090 on prefill, and it's a passively cooled server card that needs a shroud fan you'll 3D-print or eBay. Fine for a patient batch-processing box; frustrating as a daily driver.
Second 3090 vs everything else at 48GB. Two 3090s at ~$1,400 embarrass the alternatives on price (a used RTX A6000 48GB still runs ~$3,000+). You need a board with two x8 slots and a 1,200W PSU, but PCIe bandwidth barely matters for inference after model load — layer-split in llama.cpp and tensor parallel in vLLM both run happily at x8/x8. For chassis, power, and what to avoid in old server iron (noise, mostly), see the off-lease hardware field guide.
Apple Silicon. The Mac proposition is capacity per dollar and per watt, not speed: a 64GB M4 Pro mini fits a 70B Q4 that no consumer GPU under $3,000 can hold, in 10W-idle silence. Bandwidth tells you the speed story — M4 Pro at 273GB/s manages ~6–8 tok/s on 70B Q4; M4 Max at 546GB/s roughly doubles that. The persistent weakness is prefill: GPU compute is the bottleneck there, so long-context RAG on a Mac means noticeable wait-before-first-token. Buy the Mac if you want a Mac anyway or need big models quietly; buy the 3090 if tokens per second is the metric.
CPU-only: the honest numbers
Decode on CPU is bound by RAM bandwidth like everything else. Dual-channel DDR5-5600 delivers ~90GB/s, which yields 10–14 tok/s real on an 8B Q4 — perfectly usable chat. A 32B crawls at 3–4 tok/s; a 70B at 1–2 tok/s is batch-only. The killer is prefill: without GPU parallelism, a 4,000-token prompt can take 30–60 seconds before the first token appears, so CPU-only suits short-prompt chat and overnight pipelines, not document work. The exception that surprises people: used servers with 8–12 memory channels (EPYC platforms reach 400+ GB/s) decode mid-size models at GPU-adjacent speeds — a legitimate niche if you already own the iron and the ear protection.
Power, noise, placement
A 3090 box idles at 60–80W total (the card alone ~20–25W), which is $80–105/year at $0.15/kWh running 24/7 — before you've generated a single token. If the machine mostly waits, configure your runner to unload models after idle (Ollama does this by default with OLLAMA_KEEP_ALIVE) and let the GPU sleep. Under load you're at 350–450W of heat and real fan noise: cupboard or basement, not desk, and the person who shares your home gets a vote.
What I'd do, by budget
$0: run 4B models on the laptop you own — current small models are genuinely useful and it costs nothing to find your actual requirements. ~$250: a used RTX 3060 12GB brings the 14B class to interactive speeds. ~$700–900: the used 3090; this is the buy for most people, opening the 32B class where local models stop feeling like toys. ~$1,500–1,800: second 3090 for 70B territory, or a 4090 if you also game. $2,000+: the Mac Studio path if silence and capacity outrank speed. Whatever you buy, spend the first evening with the Ollama model library and a task-specific eval rather than a benchmark chart — the right hardware is the tier where the model that does your job fits, and one tier of headroom above it.