Users forgive a slow completion; they do not forgive a slow start. The budgets that hold up in practice: an inline suggestion has 150–400ms total; a chat response has about one second to first token; anything silent past three seconds reads as broken. Once tokens are flowing, the bar drops to reading speed — people read at 4–6 words per second, so 20+ tokens/s feels smooth and 60 tokens/s is indistinguishable from 200. The engineering consequence: spend almost everything on time-to-first-token (TTFT) and almost nothing on generation speed beyond ~30 tokens/s.
Budgets by surface
| Surface | Budget | The number that matters |
|---|---|---|
| Inline autocomplete | 150–400ms total | Full response — no streaming can save you |
| Chat / assistant | ≤1s TTFT, ≥20 tok/s | TTFT |
| RAG answer | ≤1.5s to first useful text | Retrieval + TTFT combined |
| Agent / multi-step | Visible progress ≤500ms | Time to first event, not first token |
| Background jobs | None user-visible | Throughput; show status, not spinners |
Two implications people miss. Autocomplete cannot use a frontier model over the public internet — the network round-trip alone eats half the budget, which is why every serious completion product runs a small model, often regional or on-device. And agents don't need fast models as much as they need fast narration: "Searching your documents…" rendered at 300ms buys you ten seconds of patience for the actual work.
Where TTFT actually goes
For a hosted API call, TTFT decomposes into network and TLS (50–150ms), provider queueing (variable, and where your p99 lives), and prefill — processing your prompt, which scales linearly with prompt length. That last term is the one you control. Levers, in order of typical impact:
Cache the prefix. Prefill dominates TTFT on any prompt past a couple thousand tokens, and cached prefixes skip most of it — on long agent transcripts I've seen TTFT drop from 3–4s to under 1s from cache markers alone. Mechanics and pricing in prompt caching economics.
Delete tokens. Every 1,000 prompt tokens is very roughly 50–150ms of prefill on hosted frontier models (estimate; it varies by provider and load). Ten-thousand-token system prompts are a latency decision, not just a cost one.
Use a smaller model for the first response. A small-first routing tier improves latency for the same reason it cuts cost, and for latency purposes even a draft response that gets refined asynchronously can be right.
Kill serial pre-calls. A moderation check or query-rewrite call that runs before the main call adds its full latency to TTFT. Run moderation in parallel and cancel generation on a hit; inline the rewrite into the main prompt. I've seen this one change cut a product's TTFT by 40%.
Keep connections warm. Persistent HTTP/2 connections and clients that don't re-handshake per request save 100–300ms of tail. Boring, real.
Streaming is mandatory, with two caveats
Stream every text surface — it converts a 6-second wait into a 700ms wait plus 5 seconds of reading. Caveat one: don't stream raw JSON at a UI that can't render partials; stream semantic events instead (status updates, completed fields as they close) and let structured payloads land whole. Caveat two: partial markdown renders badly — tables and code fences flicker between broken and fixed states — so buffer at block boundaries, not token boundaries. Both caveats argue for an event protocol between model and UI rather than piping the token stream straight through.
Prefetch: spend cents to hide seconds
The cheapest perceived-latency wins come from starting work before the user asks. Fire the retrieval query on a 300ms typing pause so RAG context is warm when they hit enter. Pre-generate the three most likely follow-ups after each assistant turn with a cheap model (this is what suggested-reply chips are) and serve instantly on match. Send a tiny request when a user opens the AI panel so the cache is warm before their first real message. Prefetch typically costs 5–15% extra tokens; against small models routed correctly, that's noise compared to the UX difference.
Hedge, then fall back
For tail latency, hedging beats retrying: if no first token has arrived by your p90–p95 mark (say 2.5s), fire a second request to another provider or a smaller model and take whichever streams first, cancelling the loser. Triggered at p95, hedging costs roughly 5% duplicate spend and can halve p99 TTFT. Below the hedge sits the hard fallback: a deadline (5–8s) after which the feature degrades honestly — a search-results list instead of a generated answer, a template instead of prose. The wiring — timeouts, cancellation, multi-provider clients — is the same machinery as surviving API failures; latency work gets it for free if reliability work came first.
And measure percentiles per surface, not means: log TTFT and tokens/s on every request in your traces (Langfuse captures both without ceremony). A 900ms mean with a 6s p99 is a product with a reputation for being slow, because users remember the p99.
What I'd do
Set the budget per surface from the table, then instrument TTFT before touching anything — most teams discover one serial pre-call and one uncached 4,000-token prefix and hit their chat budget in a week. Order of operations: streaming on, serial calls parallelised, prefix cached, then routing, then hedging at p95, then prefetch as polish. Skip generation- speed work entirely until every surface streams above 25 tokens/s from its cheapest adequate model — past that, no user can tell.