Context Window Management: Budgeting, Caching, Compaction

Million-token context windows didn't end context management; they moved the problem from "does it fit" to "what does it cost". Every turn of a conversation or agent loop re-sends the entire history, so context size is a multiplier on your bill and your latency, and long-context attention quality degrades well before the hard limit. Three practices keep it under control: a written token budget per feature, cache-first prompt layout with the stable content up front, and pruning tool results before you summarise anything.

Set a token budget like a performance budget

Write down what each request is allowed to contain, per feature, the way you'd budget page weight. A working example for a RAG chat feature:

The budget's value is that violations become visible. Context creep is how LLM features die economically: someone raises top-k from 5 to 12, someone appends a "helpful" preamble per turn, and six weeks later the same feature costs 3× per request with no quality gain anyone can demonstrate. The budget line in your dashboard is what catches it — the same discipline as cost engineering generally, applied at the request level.

Cache-first layout: stable prefix, volatile tail

Provider prompt caching is prefix-matching: the request is cached up to the point where it first differs from a previous one, and everything after the first differing byte is full price. Cache reads are cheap — roughly 10% of the input price on explicit-caching APIs, with a modest write premium; automatic-caching providers discount around half. At agent scale this is the difference between a viable feature and an absurd bill, and you get it by ordering the request correctly:

request = [
    system_prompt,      # frozen at deploy — no date, no user name
    tool_definitions,   # stable order, deterministic JSON
    # ---- cache boundary: everything above identical across requests ----
    rolling_summary,    # changes rarely
    history[-10:],      # recent turns verbatim
    retrieved_context,  # per-request
    user_message,       # per-request
]

The classic self-inflicted wounds are all one line long: f"Today is {datetime.now()}" in the system prompt, a UUID in the header, tools serialised from an unordered dict. Each one makes every request a cache miss, invisibly. Verify with the usage fields your API returns — if cached-token counts sit at zero across identical-prefix requests, something upstream is mutating the prefix. And when an instruction has to change mid-session, append it as a late-position message rather than editing the system prompt; an edit at position zero re-prices the entire conversation. This is also why prompt review should treat edit position as a first-class concern.

Prune before you summarise

When history grows, the instinct is summarisation. Do the cheaper thing first: drop old tool results. In agent transcripts they're typically 60–80% of the tokens, and a 2,000-token search result from twelve turns ago has almost always served its purpose — the model extracted what it needed at the time. Replace stale tool outputs with a one-line tombstone ([tool result elided: 1,987 tokens, web_search "pgvector hnsw params"]) and keep the tool calls so the trajectory stays legible.

Then, and only then, summarise: a rolling summary of everything older than the last ~10 turns, with recent turns kept verbatim. Summarisation is lossy in ways pruning isn't — names, numbers, and constraints vanish — so pin anything that must survive (the original task statement, hard requirements, decisions taken) outside the summarised region. Several APIs now offer server-side compaction that does this management for you; same idea, managed — worth taking when offered, but the pin-what-matters rule still applies.

Long context degrades before it overflows

Two facts justify aggressive management even under huge windows. First, models attend unevenly across long contexts — the "lost in the middle" effect (Liu et al.) where information buried mid-context is recalled worse than the same information at either end. Current models are better than the 2023 ones that paper measured, but the gradient hasn't vanished: put instructions and the question near the edges, and don't bury the one critical constraint at token 60,000. Second, stuffing beats retrieval only at small scale. "Just paste the whole wiki" is tempting under a 1M window and wrong at both ends: you pay for every token on every turn, and answer quality on needle-ish questions drops relative to retrieving the five relevant chunks. Big windows are for genuinely long artifacts — codebases, transcripts, contracts — not a substitute for search.

Measure it or lose it

Minimal instrumentation that pays for itself immediately: per-feature p50/p95 input tokens, cache-hit ratio, and context growth per session turn. Alert on step changes — a cache-hit ratio that fell off a cliff on Tuesday is a prompt edit that broke the prefix; input tokens climbing 2% a week is context creep. For agent loops, also track tokens-per-task-completion: it's the metric that pruning and budgets actually move, and the one that tells you whether a "smarter" prompt was worth its weight.

What I'd do

Write the budget table for your top feature today — the numbers force every subsequent decision. Then: freeze the system prompt and tool order, move volatile content to the tail, add tombstone-pruning for tool results older than ~10 turns, and put input-tokens and cache-hit-ratio on the dashboard next to latency. Summarisation, compaction features, and clever memory schemes come after those basics, and on most workloads they turn out to be optional once the basics are done.