LLM inference has two phases with opposite bottlenecks, and every optimisation in this post attacks one of them. Prefill (processing the prompt) is compute-bound; decode (generating tokens) is memory-bandwidth bound, because every new token re-reads all the weights. That second fact sets a hard ceiling you can compute on a napkin: bandwidth divided by model bytes. An RTX 3090 moves 936GB/s; a 32B model at Q4 is ~19GB; so single- stream decode tops out near 49 tok/s theoretical and 30–35 real, and no software setting changes that. Everything past the ceiling is either batching (throughput), a smaller memory footprint (quantisation), or getting multiple tokens per weight-read (speculative decoding).
Where the time goes
Time-to-first-token is prefill plus queueing: roughly linear in prompt length and parallelisable, so GPUs chew through it at thousands of tokens per second. Per-token latency after that is decode, and it barely depends on prompt length until the KV cache stops fitting. The practical implications invert most people's instincts: long prompts hurt TTFT but not generation speed; a "slow model" complaint with fast first tokens is a decode-bandwidth problem; and prefill and decode fight for the same GPU, so one user's 40k-token prompt stalls every other stream's token flow unless the server interleaves them (chunked prefill, below).
Continuous batching is the biggest single win
Decode wastes the GPU: one stream reads 19GB of weights to produce one token. Batch 32 streams and you read the weights once per step for 32 tokens — decode throughput scales nearly linearly with batch size until compute or cache memory pushes back. Continuous (in-flight) batching, which admits and retires requests every step instead of waiting for a full batch to drain, is why vLLM reported order-of-magnitude (up to ~24×) throughput gains over naive per-request serving when it launched, and why every serious server since has copied it, along with PagedAttention — block-allocated KV cache that eliminates the fragmentation which used to waste 60–80% of cache memory.
The trade is per-user latency: each stream's tok/s degrades as batch grows. Serving is choosing a point on that curve — a chatbot wants moderate batches and ~20+ tok/s per user; an offline enrichment pipeline wants the biggest batch that fits and doesn't care that each stream crawls.
KV cache math decides your concurrency
Per token, the cache costs 2 × layers × kv_heads × head_dim × bytes (the 2 is keys plus values). Llama-3.1-8B with GQA (32 layers, 8 KV heads, 128 head dim, fp16) is ~131KB per token — an 8k-token conversation holds about 1GB, so a 24GB card serving the 5GB Q4 model fits roughly 19 such conversations before evictions begin. That arithmetic, not compute, is usually what caps concurrent users; the full worked examples live in VRAM math.
The levers, in the order I'd pull them: FP8/int8 KV cache quantisation (halves cache size, quality cost small and measurable — test it like any quantisation decision); prefix caching, so the system prompt and shared document prefixes are computed once and reused across requests (huge for agent workloads that re-send context — structure prompts static-first to exploit it); and sliding-window or cache-eviction schemes, which trade long-range recall for memory and need eval coverage before you trust them.
Speculative decoding: spend compute to save latency
A small draft model proposes k tokens; the big model verifies them in one parallel pass — one weight-read for potentially several accepted tokens, with output provably identical in distribution to the target model. Acceptance rate decides everything: on code and formulaic text, drafts hit 70–80% acceptance and you see 1.8–2.5× decode speedups; on creative prose, acceptance drops and gains thin toward zero. Self-speculative variants (EAGLE/Medusa-style heads) avoid running a second model and ship built into the major servers now. The catch: speculation spends spare compute, so its benefit collapses at high batch sizes where compute is already saturated. It's a low-batch, latency-sensitive tool — ideal for a self-hosted single-user assistant, mostly pointless on a saturated production server.
Chunked prefill and the p99 you're ignoring
On mixed workloads the ugliest latency number is inter-token p99, and its usual cause is head-of-line blocking: a monolithic 40k-token prefill freezing every decode stream for hundreds of milliseconds. Chunked prefill splits prompt processing into slices interleaved with decode steps, trading a little TTFT on the long-prompt request for smooth token flow on everyone else's. If you serve both RAG (long prompts) and chat (long generations) on one deployment, turn it on and re-measure; it's the single most common fix for "generation stutters under load".
Pick the stack by workload
| Stack | Built for | Reach for it when |
|---|---|---|
| vLLM | Throughput serving, continuous batching, PagedAttention | Multi-user production on NVIDIA GPUs |
| SGLang | Agent/structured workloads, radix prefix cache | Heavy shared-prefix traffic, constrained output |
| llama.cpp | Single user, GGUF, CPU/Mac/consumer GPU | Self-hosting, edge, heterogeneous hardware |
| TensorRT-LLM | Peak NVIDIA performance | You have an infra team and stable models |
The single-user-vs-server decision is the big fork — the vLLM vs llama.cpp comparison works through it with benchmarks and a methodology you can rerun on your own hardware.
What I'd do
Compute your bandwidth ceiling first; it calibrates every expectation. Self-hosting for yourself: llama.cpp or Ollama, Q4, speculative decoding on, done. Serving users: vLLM with continuous batching, prefix caching on, chunked prefill on, FP8 KV cache after an eval pass, batch size chosen by watching per-user tok/s against your latency budget. Measure TTFT, per-user tok/s, and aggregate tok/s separately — every good decision in inference is a knowing trade between those three, and every bad one comes from optimising whichever single number someone happened to graph.