LLM spend is the one infrastructure line that can 10× in a week without anyone deploying anything, and the fix is unglamorous: per-feature unit costs, a dashboard, and five levers applied in order — prompt caching, model routing, output-length control, batch APIs, and context pruning. Applied to a typical assistant-style product, that sequence cuts 50–80% of spend. In every bill I've helped audit, the money was hiding in output tokens and retry loops, not in the model's list price.
You can't engineer what you can't attribute
Step zero is tagging every API call with feature, user or tenant, and environment, then computing unit cost: (input tokens × input price + output tokens × output price) per request, rolled up per feature per day. Without attribution you get the classic failure: the bill doubles, and it takes three days of log spelunking to learn that a background summarisation job got looped. Self-hosted Langfuse does this well — traces carry metadata, and cost-per-feature is a saved view. The numbers that should be on one screen: daily spend by feature, unit cost by feature, cache hit rate, and tokens per request p50/p99. An example of what attribution surfaces:
| Feature | Requests/day | Unit cost | Daily spend | Smell |
|---|---|---|---|---|
| Chat assistant | 40,000 | $0.011 | $440 | Cache hit rate 22% — layout bug |
| Auto-title | 60,000 | $0.004 | $240 | Frontier model for 8-word outputs |
| Nightly digest | 3,000 | $0.09 | $270 | Real-time API for a cron job |
Every row of that table is a lever with a name on it.
The five levers, ranked
1. Prompt caching — 30–70% off input spend, zero risk. Reorder prompts so static content precedes volatile content and add cache markers where explicit. No quality trade-off, no eval needed, one PR. Details and provider math in prompt caching economics.
2. Model routing — 40–70% off overall, needs an eval. Most traffic does not need the model it's getting. Static per-feature rules first (the auto-title row above is a one-line fix for ~20× savings on that feature), learned routing second. The harness requirements are the real cost — see send small models first.
3. Output-length control — 20–50% off output spend. Output tokens cost 3–5× input tokens and dominate generation latency. Set max_tokens per feature instead of one global 4,096; instruct brevity concretely ("3 bullet points, no preamble" beats "be concise"); use stop sequences; return structured fields instead of prose where the consumer is code; and don't ship extended reasoning tokens on tasks that don't score better with them — reasoning modes are an output-token firehose and several providers bill them as output even when hidden.
4. Batch APIs — flat 50% on everything that can wait. Both Anthropic and OpenAI price batch at half of real-time. Anything cron-shaped — digests, enrichment, evals, embeddings backfills — is overpaying by 2× on the real-time endpoint. The nightly-digest row above becomes $135/day by changing an endpoint. Architecture in batch vs real-time inference.
5. Context pruning — 10–40% off input, apply last. Trim retrieval to what moves answers (5 chunks is usually indistinguishable from 12), cap conversation history, compact agent transcripts at coarse checkpoints. It's ranked last because it fights lever 1 — continuous rewriting churns the cached prefix — and because it's the only lever that can quietly hurt quality, so it needs the same eval discipline as routing.
The multiplier nobody budgets: calls per task
Token prices are the visible factor; call count is the silent one. An agent that plans, calls three tools, and self-critiques is 5+ model calls per user action; add one retry layer with three attempts and your naive estimate is off by an order of magnitude. Controls that matter: cap agent iterations (a hard ceiling of 6–10 steps catches most runaway loops), make retries budget-aware rather than count-based, and log calls-per-task as a first-class metric next to unit cost. When unit cost spikes, it's almost always this number, not token inflation.
Guardrails: alerts before autopsies
A cost system needs tripwires, not just dashboards. Three that earn their keep: a daily budget alert per feature at 2× trailing-week median (catches loops within hours); per-user and per-tenant token rate limits (catches abuse and the one customer scripting your chat endpoint); and a circuit breaker that degrades or disables a feature at 3–5× its daily budget — wired the same way as your kill switches for quality incidents. Cheap models make runaway loops cheap too, but only the tripwire makes them short. Log the model name per request while you're at it — "temporary" frontier-model experiments have a way of becoming permanent line items that nobody remembers approving, and the dashboard is where you catch them.
What I'd do
Week one: attribution and the dashboard — no optimisation until the table above exists for your product, because you'll optimise the wrong feature otherwise. Week two: caching (the PR is small) and the static routing rules for your two most obviously over-modelled features. Week three: max_tokens audit and move everything cron-shaped to batch. Then stop and re-measure; in my experience you're down 50–70% and the remaining levers need eval infrastructure to apply safely. What I would not do: start with fine-tuning or provider negotiations. Both have their place — distilling a small model beats routing at high volume on a single dominant task — but they're month-three projects that look tempting mainly when the week-one measurement is missing.