Surviving LLM API Failures: Timeouts, Fallbacks, Queues

Plan for each LLM provider to deliver roughly 99.5% availability, not 99.99% — the public status pages of every major provider record multiple multi-hour degradations per year, and 99.5% is over 40 hours annually. The resilient shape is boring and specific: three separate timeout clocks, a retry policy that distinguishes 429 from 400 from a mid-stream stall, a second provider behind a thin adapter, and a queue for everything that isn't a human waiting on a spinner. None of it is novel distributed-systems work; the LLM-specific part is where the clocks go and what you must never retry.

Timeouts: three clocks, not one

A single request timeout is wrong for streaming inference in both directions: too short kills healthy long generations, too long leaves users staring at a spinner during an incident. Run three clocks per request:

TTFT is the clock that matters operationally, because provider incidents overwhelmingly present as queueing: the connection succeeds and then nothing arrives. A 15-second TTFT timeout converts "the app hung" into "the app failed over" — and TTFT p95 is the metric to alarm on, since it degrades minutes before error rates move. With streaming, "the request succeeded" is a process, not an event; instrument all three clocks separately or you'll tune the wrong one.

The retry taxonomy

Retrying everything is how a provider incident becomes your cost incident and their thundering herd. The policy that survives contact:

• 429: honour Retry-After if present; otherwise exponential backoff with full jitter (the AWS Builders' Library piece remains the canonical treatment), max 2–3 attempts. If you hit 429s steadily, retries are the wrong tool — your concurrency is misprovisioned against your rate limit (fix the token bucket, below).
• 500/502/503 and provider-specific "overloaded": one or two retries with jitter, then fail over. These cluster during incidents; cap retries hard so failover happens in seconds, not after a 60-second backoff ladder.
• 400/401/403 and schema validation errors: never retry. The request is wrong; the retry is wrong the same way. Log loudly — these are bugs, not weather.
• Mid-stream stall or disconnect: retry only if nothing was shown to the user or the operation is idempotent. Resuming a half-rendered answer by re-sending and splicing is a bug farm; prefer restart-and-replace in the UI.
• Agent tool steps: never blindly re-run a step whose tool call may have executed. Side-effecting tools need idempotency keys before retries are legal anywhere in the loop — the tool calling patterns post covers the mechanics.

Fallbacks without the lowest common denominator

The trap in multi-provider design is abstracting to the features every provider shares, which discards exactly the features you chose your primary for. Instead: a thin adapter per provider plus explicit capability flags — supports_strict_json, supports_parallel_tools, max_output_tokens — and degrade features, not requests. If the fallback lacks strict schema enforcement, the adapter switches on your validate-and-reprompt path from the structured output playbook; if it lacks parallel tool calls, the agent runs sequential. Users get a slightly slower answer instead of an error page.

Prompts are not portable between model families, and pretending otherwise turns failover into a silent quality incident. Maintain tested per-provider variants of your top prompts only — for most products that's three to five prompts carrying 90% of traffic — and accept a measured quality dip on the long tail. Tag every response with the provider that served it, and grade sampled failover traffic offline; "how much worse is the fallback" should be a number you know before the incident, which is a two-hour job with the eval harness you already have.

Trip the failover with a standard circuit breaker per provider (error rate or TTFT-p95 threshold over a sliding window, half-open probes to recover). During brownouts rather than blackouts, degrade before failing over: shorter max_tokens, your smaller model tier, non-essential AI features off first — the same ladder your flag system already implements, tripped automatically.

Queues for everything without a human waiting

The cheapest resilience is deciding which requests are actually interactive. Embedding pipelines, nightly summaries, enrichment, re-indexing — none of it needs a synchronous call, and all of it belongs on a queue with a worker pool whose concurrency is governed by a token bucket on tokens-per-minute, not requests-per-minute (LLM rate limits are TPM limits; ten small requests and one 100k-token request are not the same spend). A queue turns a two-hour provider outage into delayed batch jobs and zero paged humans.

For truly bulk work, the batch endpoints most providers now offer run at roughly half price with a 24-hour completion window — resilience and a discount from the same architectural decision. The batch-vs-realtime split is worth designing early, because retrofitting queues under incident pressure is miserable.

What I'd do

Defaults to steal: 15s TTFT, 15s inter-token, 90s total. Retries: 2 max with full jitter, 429/5xx only, Retry-After honoured, nothing retried after tokens reach the user. One fallback provider behind capability-flagged adapters, per-provider variants for the top five prompts, graded quarterly. Circuit breaker on TTFT p95 and error rate; brownout ladder before blackout failover. Queues with TPM-bucketed workers for every non-human request, batch APIs for bulk. Then run a game day: block the primary provider's endpoint at the proxy for an hour in staging and watch what actually happens — every team that does this finds one surprise, and it's cheaper to meet it on a Tuesday afternoon than during the real thing.