An AI feature is not shipped until it has four controls: a kill switch that degrades to a defined fallback, a percentage rollout, a pinned model-and-prompt version you can change without deploying code, and a cost circuit breaker. LLM dependencies fail differently from libraries — providers have multi-hour incidents, models get deprecated on a vendor's schedule, and an innocuous prompt edit can double token spend or format-failure rates overnight. All four failure types will visit you within the first six months; the flags decide whether each one is a config change or an emergency deploy.

Flag the model and prompt, not just the feature

The unit behind the flag is not "summaries on/off". It's the whole inference configuration, treated as a deployable artifact:

{
  "assistant_summary": {
    "enabled": true,
    "rollout_pct": 25,
    "model": "claude-sonnet-4-5",
    "prompt_version": "summary-v14",
    "max_output_tokens": 512,
    "timeout_ms": 20000,
    "daily_budget_usd": 40,
    "fallback": "extractive"
  }
}

Everything in that blob changes more often than your code does, and every field has caused a production incident somewhere: the model id (deprecation), the prompt version (quality regression), the token cap (cost), the timeout (provider latency shift). Putting them behind your flag system — OpenFeature with any backend, LaunchDarkly, Unleash, or a table you poll every 30 seconds — means each incident resolves in seconds. Prompt versions belong in version control with tests, exactly as argued in prompt engineering is just engineering; the flag selects among versions, it doesn't contain the prompt.

Shadow mode before anyone sees anything

Run the feature silently on real traffic first: execute the inference on a sampled percentage of requests, log everything, show users nothing. Two weeks or ~1,000 samples per meaningful segment, whichever is later. Shadow mode is the only honest way to learn the numbers that decide launch: real p95 latency (your staging prompts are shorter than production ones — everyone's are), cost per request against real inputs, format-failure and refusal rates, and how outputs compare to the current non-AI behaviour. Half the AI features I've shadowed changed materially before launch — usually the token cap and the timeout — and one never launched, because shadow mode priced it at 9× the estimate. That's the cheapest failed launch you'll ever have.

Score the shadow outputs with the same graders you use offline; if you don't have graders, you're not ready to shadow, and evals before vibes explains what to build first.

Kill switches that actually kill

A kill switch is only real if three things are true. It flips server-side within seconds, no deploy and no app-store release in the path. It degrades to a defined fallback, decided when the feature was designed: AI summary falls back to extractive first-paragraphs, semantic search to keyword search, the smart-reply button to not existing — hiding a feature cleanly is a perfectly good fallback, a spinner that never resolves is not. And it gets tested: flip every kill switch in staging on a schedule (monthly is fine), because an unexercised kill path rots like an untested backup. Wire the switch to the same degradation logic your API-failure handling uses — a provider outage and a deliberate kill should walk the same code path, which means the provider outages are rehearsing your kill switch for free.

Cost circuit breakers, with real numbers

Token spend is the failure mode unique to AI features, and it fails fast: a retry loop against a slow provider, one user scripting your endpoint, or a prompt change that triples input size can turn $40/day into $400/day between dashboard glances. Budgets go at two levels. Per-user daily caps (for a summary feature, something like 50 requests or $0.50/day) stop abuse and runaway automation. Per-feature daily budgets stop systemic surprises: alert at 60% of budget, degrade at 90% — first to a cheaper model, then to the fallback — and never simply 500. The math takes a minute: 2k input + 500 output tokens at a mid-tier model's rates is roughly $0.01/request, so 10,000 requests/day is $100/day and your budget line writes itself. Track cost per request per feature in your tracing stack (Langfuse does this out of the box) so the breaker's denominator is measured, not guessed.

Gradual model swaps are the muscle you'll use most

Providers deprecate models on 6–12 month cycles, so model migration is not an if. The routine, once per swap: add the new model as a flag arm at 5%, compare graded outputs and the operational numbers (latency, cost, format-failure rate) against the incumbent arm on live traffic, then walk 5% → 25% → 50% → 100% over one to two weeks. Keep the old arm configured and warm for 30 days after — instant rollback is the whole point — and only delete it when the new arm has survived a full traffic cycle including whatever your weekly peak looks like. Prompts rarely transfer unmodified between model families; budget a prompt-version bump as part of every swap rather than discovering it at 50%.

The metrics that gate promotion

Promote a flag stage only on numbers: format-failure rate (should be near zero if you're using structured outputs), p95 latency against the budget you set, cost per request, explicit user signals (thumbs, regenerate-rate, edit-distance between the AI draft and what the user actually kept), and task completion downstream. Regenerate-rate is the quiet workhorse — users who click regenerate twice are telling you the output is bad more honestly than any survey.

What I'd do

One config artifact per AI feature holding model, prompt version, caps, budget, and fallback, served by your flag system. Shadow mode for two weeks with graded outputs before any user sees the feature. Kill switch flipped monthly in staging. Per-user and per-feature budgets with a degrade-then-disable ladder at 60/90%. Model swaps as a rehearsed 5→100% routine with 30 days of instant rollback. None of this is exotic infrastructure — it's the same flag discipline you already apply to risky code, extended to the parts of an AI feature that change under you: the model, the prompt, and the bill.