Prompt caching is a 50–90% discount on input tokens and a 2–4× cut in time-to-first-token, with zero quality trade-off, and most teams leave it on the table because their prompt layout defeats it. If every request ships a 3,000-token block of system prompt, tool definitions, and few-shot examples, you are paying full price to re-process identical bytes thousands of times a day. The fix is usually an afternoon of reordering, not a rewrite.
How the providers differ
All the major APIs cache the key-value state of a prompt prefix — the longest initial span of tokens identical to a previous request. The economics differ enough to matter (numbers as of mid-2026; confirm against the pricing pages before doing your own math):
| Provider | Mechanism | Read price | Write premium | Min prefix | Lifetime |
|---|---|---|---|---|---|
| Anthropic | Explicit cache_control markers | 10% of base input | +25% (5-min) or +100% (1-hour) | 1,024–2,048 tokens | 5 min, refreshed on hit; 1h option |
| OpenAI | Automatic on repeated prefixes | 50% of base (better on some models) | none | 1,024 tokens | ~5–10 min, longer off-peak |
| Google Gemini | Implicit + explicit modes | 25% of base | none implicit; hourly storage fee explicit | ~1,024–2,048 tokens | varies / you choose |
| DeepSeek | Automatic, on by default | ~10% of base | none | 64-token blocks | hours |
The Anthropic prompt caching docs and OpenAI's guide cover the mechanics. Anthropic's write premium changes the math slightly: a 5-minute-TTL cache write costs 1.25× base, so one reuse within the window already nets you 1.35× total spend versus 2.0× uncached — a 32% saving from a single hit, compounding from there.
Layout: static first, volatile last
Caching matches from token zero. One dynamic token near the top — a timestamp, a request ID, the user's name — invalidates everything after it. The canonical order:
1. System prompt (changes per deploy)
2. Tool/function definitions (changes per deploy)
3. Few-shot examples (changes per deploy)
4. Slow-moving context (user profile, workspace docs)
5. Conversation history (grows per turn)
6. Current user message (unique per request)The classic self-inflicted wounds, all of which I have shipped or reviewed: Today's date: 2026-02-08 14:32:11 as line one of the system prompt (round to the day and move it down); retrieval results injected above the static few-shots; JSON tool schemas serialised from a dict with nondeterministic key order, so byte-identical intent produces different tokens; and an A/B framework that interleaves prompt variants per request, halving the hit rate of both arms.
Agents are the killer use case
A multi-turn agent resends the entire transcript every turn: system prompt, tools, and all prior tool calls and results. Uncached, an agent that takes n turns pays roughly O(n²) in input tokens across the episode. With the prefix cached and each turn appending to it, every turn re-reads the whole history at the discounted rate — on Anthropic that's 90% off nearly all of the episode's input volume. On agent-heavy workloads I've measured input spend dropping 60–75% from cache markers alone, before any other optimisation. The same effect cuts prefill latency, which is most of time-to-first-token on long transcripts — relevant to your latency budgets as much as your bill.
Measure the hit rate or you're guessing
Every provider reports cache activity in the usage block — cache_read_input_tokens on Anthropic, cached_tokens inside prompt_tokens_details on OpenAI. Define hit rate as cached input tokens ÷ total input tokens and log it per feature, not globally; a healthy multi-turn chat feature should sit above 70%, a single-shot classification endpoint with a fat shared system prompt above 50%. Pipe the usage fields into whatever traces you already keep — self-hosted Langfuse breaks out cached versus fresh input per request, which makes regressions visible the day a deploy reorders the prompt.
Two structural reasons for a low hit rate that aren't layout bugs. Low traffic: a cron job that fires hourly against a 5-minute TTL hits nothing — either batch its work into one burst, pay for Anthropic's 1-hour TTL, or accept the miss. High cardinality: if each of 10,000 users has a unique 4,000-token profile block, you get per-user caches that only pay off for active users mid-session — which may still be most of your volume, but check before celebrating.
Where caching ranks among cost levers
First. It requires no model change, no quality evaluation, no user-visible difference, and it stacks with everything else — routing, batching, output control. In the cost-lever ranking it's the only one that is nearly pure upside; the runner-up, routing to smaller models, needs an eval harness before you can trust it. Context trimming is the one lever that fights it: aggressively rewriting or compacting history churns the prefix and kills the cache, so compact at coarse boundaries (every 20 turns, at a summary checkpoint), not continuously — more on that trade-off in context window management.
Bottom line
Reorder your prompt so everything static precedes everything volatile, add explicit cache markers if you're on Anthropic, then watch the usage fields for a week. For a typical assistant-style workload this is a 30–60% cut in input spend and a visibly snappier first token, delivered in one PR. If your hit rate is still under 40% after the reorder, the cause is TTL or cardinality, not layout — fix the traffic shape or buy the longer TTL, and only then spend effort on fancier levers.