Model Routing: Send Small Models First

In most production LLM apps, 60–80% of requests can be handled by a model that costs 10–30× less than the frontier model answering them today, and routing is how you collect that money. The catch is that a router is a model-quality decision made per request, which means it needs an eval harness, calibration, and drift monitoring like any other model — and in most deployments I've seen, it gets none of the three. Route small-first, but evaluate the router or you're shipping silent quality regressions.

The economics in one table

Current price spreads make the case by themselves. Illustrative mid-2026 numbers for a request with 2,000 input and 400 output tokens:

Route 70% of traffic to the small tier and escalate the rest: blended cost is 0.7 × $0.00054 + 0.3 × $0.012 ≈ $0.0040 — a 67% cut without touching the hard 30%. On a $30k/month bill that's $20k, which buys a lot of eval harness. Routing is the second-biggest lever in LLM cost engineering after prompt caching, and unlike caching it compounds with volume growth on the expensive tier.

Three router designs, in the order to try them

Static rules first. Feature and task type are the strongest routing signal you have, and they're free: autocomplete, title generation, and summarise-this-thread never need a frontier model; multi-step agent planning always does. Most teams can move 40%+ of volume with an if-else on feature name plus input length. Do this before anything learned.

A trained classifier second. Embed the request (any cheap embedding model), train logistic regression or a small gradient-boosted model on labels of the form "was the small model sufficient here?". You get labels by running 2–5k historical requests through both models offline and having a judge model compare outputs against a rubric. Inference overhead is 5–20ms and fractions of a cent. This is the RouteLLM shape, and it's the sweet spot for mixed chat traffic.

A cascade third. Small model answers; a verifier (self-assessment, a judge, or task-specific checks like schema validation and unit tests) accepts or escalates to the big model. Cascades capture the most savings on tasks with cheap verification — code with tests, extraction with schema checks — and the least on open-ended prose, where the verifier is as hard as the task. Mind the latency: every escalation pays the small model's full latency before the big model starts, so cascade only where your latency budget absorbs a failed first try, or hedge by streaming the small answer while the verifier runs.

Calibrate thresholds; never trust raw confidence

Asking the small model "rate your confidence 0–10" and thresholding at 7 is not calibration, it's theatre — small models are systematically overconfident, and the bias shifts per task. Calibration is empirical: on a labeled set, plot the small model's actual win rate against whatever confidence signal you're using (self-score, judge score, verifier margin, classifier probability), then pick the threshold where routed quality meets your target. Expect the curve to be ugly. A workable recipe: require the classifier's P(small sufficient) > 0.8 for silent routing, send 0.5–0.8 to the cascade with verification, and everything below straight to the frontier tier. Recalibrate whenever either model changes — a small-model upgrade shifts the frontier as much as a frontier swap does.

Evaluate the router like a model

The router needs its own golden set: a few hundred requests labeled with "small sufficient: yes/no", stratified across features and difficulty. Track three numbers weekly. Routed quality delta: overall quality of the routed system versus an all-frontier baseline on the same set — keep it under 2% on your task metric or the savings aren't free. Escalation rate by feature: a rising rate means traffic drifted or the small model regressed; a falling rate with flat quality means you can loosen the threshold. Shadow disagreement: sample 1–5% of small-routed production traffic, run the frontier model on it too, and have a judge compare — this is your drift alarm, and at 1% sampling it costs about 1% of the money the router saves. The judging mechanics are the same as any LLM eval harness; the only new part is that the unit under test is the routing decision.

Failure modes I've actually seen

New intents default to the small model because the classifier was trained before the feature existed — add a "route unknown to frontier" rule and retrain monthly. A cascade that doubles p95 latency because 40% of requests escalate after a full small-model generation. A 7B router in front of a $0.15/M target model, spending more on routing than routing saves — routers must be at least an order of magnitude cheaper than the cheap path. And the quiet one: nobody owns the router, so eighteen months later nobody knows why 22% of traffic escalates, and the thresholds are archaeology. Put the router config in version control next to the prompts.

What I'd do

Ship static rules this week — feature-based routing needs no ML and no courage. Then build the labeled set (2k requests, both models, judge comparison), train the embedding classifier, and turn it on for the top two traffic features with a 1% shadow sample as the tripwire. Hold the cascade pattern in reserve for verifiable tasks only. If your monthly spend is under ~$2k, skip all of it except the static rules — the eval infrastructure costs more than it saves, and distilling a small model for your one dominant task is probably the better project.