AI Optimisation
Inference, cost, and performance engineering for AI systems. 15 posts.
VRAM Math: What Actually Fits on Your GPU
Weights plus KV cache plus overhead — the arithmetic that predicts what fits on your GPU, with worked examples from 8GB laptops to 24GB cards.
AI OptimisationvLLM vs llama.cpp: Pick by Workload, Not by Hype
vLLM wins concurrent throughput, llama.cpp wins on modest hardware and single streams. Where the crossover sits, plus a benchmark method you can rerun.
AI OptimisationQuantisation Explained: GGUF, AWQ, and What You Actually Lose
GGUF, AWQ, and FP8 quantisation explained — size and quality by bit depth, what degrades first, and how to A/B a quant honestly before trusting it.
AI OptimisationPrompt Caching: The Cheapest Optimisation You're Not Using
Prompt caching cuts input token cost by up to 90% and slashes TTFT. Cache-friendly prompt layout, provider differences, and measuring your hit rate.
AI OptimisationOllama Beyond the Laptop: Serving Models Properly
Ollama's defaults are laptop defaults. The env vars that make it serve a team — parallelism, keep-alive, KV cache — and the point where vLLM takes over.
AI OptimisationModel Routing: Send Small Models First
Most LLM traffic can run on a model 20x cheaper. Cascade designs, calibrated confidence thresholds, and how to evaluate the router like a model.
AI OptimisationLoRA Fine-Tuning on a Consumer GPU
QLoRA fine-tunes an 8B model on a 24GB GPU in an evening. What fits at each size, the hyperparameters that matter, and the evals to run before and after.
AI OptimisationLocal AI and Privacy: What Actually Stays on Your Machine
A packet-capture audit of Ollama and Open WebUI — prompts stay local, update checks don't. How to verify egress yourself and run truly offline.
AI OptimisationLLM Inference Optimisation: Batching, KV Cache, Speculative Decoding
Where LLM inference latency actually goes — prefill vs decode, KV cache maths, continuous batching, and speculative decoding, with numbers per stack.
AI OptimisationLLM Cost Engineering: Treat Tokens Like a Budget Line
Unit costs per feature, a cost dashboard, and the five levers ranked — caching, routing, output control, batch, context pruning. Where LLM bills go.
AI OptimisationLatency Budgets for AI UX: Streaming, Prefetch, Fallbacks
Users forgive slow completions, not slow starts. Latency budgets by surface, time-to-first-token engineering, prefetch tricks, and hedged fallbacks.
AI OptimisationHardware for Self-Hosted LLMs: From 8GB Laptops to 80GB Rigs
What runs at each VRAM tier from 8GB laptops to 80GB rigs — used-market GPU picks like the $700 RTX 3090, plus an honest CPU-only reality check.
AI OptimisationDistillation: 80% of the Quality at 5% of the Cost
A small model trained on filtered frontier outputs can hit 90% of teacher quality on narrow tasks. The pipeline, the licensing traps, break-even math.
AI OptimisationBenchmarking Local LLMs Without Fooling Yourself
Contamination, quant variance, and template bugs move scores more than model choice. Build a 100-item harness from your own data and run it tonight.
AI OptimisationBatch vs Real-Time Inference: Two Different Systems
Batch APIs cost half of real-time and finish within hours. What belongs in each system, the queue architecture batch needs, and hybrid designs.