Send the easy ones
somewhere cheap.

Production LLM traffic is lopsided: a small fraction of requests are genuinely hard (multi-step reasoning, tricky code, ambiguous instructions), and the rest are easy. Yet the default architecture sends all of it to one big, expensive model. Model routing replaces that flat policy with a tiered one — match each request to the cheapest model that can actually handle it — echoing the same idea that makes mixture-of-experts efficient: don't activate the whole giant when a small part will do.

Because price gaps between model tiers are large (a small model can be 10–100× cheaper than a flagship), moving even a modest share of traffic to a cheaper tier is a big win. And since the easy majority is exactly where cheap models are already good enough, routing usually cuts cost sharply while leaving quality almost untouched. The whole discipline reduces to one question: how do you decide which model gets each request, and what does that decision cost?

The most common routing design is a cheap-smart cascade. Every request goes to the cheap model first. Then you check whether the cheap answer is trustworthy — via a confidence score, a quick self-check, or a small verifier — and only when it's unsure do you escalate the request to the expensive model. Confident cheap answers are returned directly; the rest fall through to the flagship.

The cascade is appealing because it's reliable: the cheap model never handles a request it flagged as hard, so mis-routing a difficult query to the cheap tier can't happen (the escalation step catches it). The price of that reliability is the cheap model's cost on every request — including the ones that end up escalating anyway, where you've paid twice. That double-pay on escalated requests is exactly what the cost math has to weigh.

Here's the counterintuitive part. If your cheap model escalates everything, the cascade costs more than just using the big model — you've paid the cheap model's price on top of the flagship's, for nothing. The cascade only saves money when the cheap model actually resolves a large share of requests, keeping the escalation rate low.

So the single number that determines success is the escalation rate — the fraction of requests that fall through to the expensive model. Everything you do to improve routing (a better confidence signal, a stronger cheap model, a smarter threshold) is really an effort to push that rate down. And how low it needs to go depends on the price gap: a much cheaper cheap-model tolerates a high escalation rate; a barely-cheaper one demands a very low one. The next section turns this into an exact break-even.

Let the cheap and big model costs per request be c_cheap and c_big, and let r be the escalation rate. The cascade pays the cheap model on all requests and the big model on the escalated fraction:

The c_cheap term is paid on every request, escalated or not — that is the cascade's overhead.

Set them equal and solve for the escalation rate at which the cascade breaks even:

If the cheap model is 1/20 the price (c_cheap=1, c_big=20), break-even is r_*=0.95 — you can escalate up to 95% of traffic and still win. If it's 1/100 the price, r_*=0.99. The bigger the price gap, the more escalation you can afford. Stay below r_* and the cascade saves money; go above and it loses. The runnable version below computes the cascade cost, the break-even, and where a cascade wins or loses.

Cascades aren't the only routing design. The main alternatives trade overhead against risk:

Predictive routers avoid the cascade's double-pay by not running the cheap model on requests destined for the flagship — but they can send a hard request to the cheap model and get a bad answer, since they decide before seeing any output. Cascades never make that mistake but pay more. In practice many systems combine them: a fast router for obvious cases, a cascade with a verifier for the rest. Either way, this is cost engineering applied to model selection — measure your escalation rate and price gap, and the right design follows.

The dangerous failure is a bad confidence signal. A cascade is only as good as its ability to know when the cheap model is wrong — and cheap models are often confidently wrong, which is the worst case: high confidence, wrong answer, no escalation. If your escalation trigger doesn't correlate with actual correctness, you'll ship cheap mistakes while thinking you're safe. Validate the signal against real accuracy, not the model's self-reported certainty.

Two more. Latency: a cascade adds the cheap model's time to every escalated request, so tail latency can rise even as cost falls — sometimes a predictive router is better for latency-sensitive paths. And cost per successful outcome is the real metric: a cheap answer that fails and triggers a retry, a complaint, or human review can cost more end-to-end than routing straight to the flagship. As with all AI cost work, measure the full-funnel cost of an outcome, not the sticker price of a call — then the escalation rate and price gap tell you exactly where the cascade pays.