The big idea

What does an LLM router do?

Not every request needs your biggest, most expensive model. "Reformat this JSON" and "reason step-by-step through this legal argument" cost the same if you send both to the flagship — but the first could be handled by a model 10–50× cheaper and faster with identical quality. Send everything to the big model and you overpay massively; send everything to the small one and hard queries fail.

An LLM router picks, per request, the cheapest model that can still get it right. It classifies the request's difficulty/type, routes easy ones to a small model and hard ones to the flagship (or a specialist), and escalates when the cheap model isn't confident. Same quality, a fraction of the cost — the routing decision is the whole system.

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the pipeline grow, and at the end blind the router and kill the strong model to see what the routing signal and escalation path buy you. Hit Begin.

Step 1 · One model for all is wasteful

The cost/quality bind

The two naive options: route everything to the big model, or everything to the small one. Why is each a bad deal?

Design decision: Send all requests to one model. Big model for all, or small for all — what's wrong with each?

The call: Both are fine; just pick one and move on. — They're not equivalent trade-offs to ignore — big-for-all can be 10–50× more expensive than necessary on easy queries, and small-for-all degrades quality where it matters. Per-request routing captures the savings without the quality loss.

Big-for-all overpays — most requests are easy and don't need flagship capability, so you burn 10–50× the necessary cost and latency on trivial work. Small-for-all fails the hard minority with weak or wrong answers. Because difficulty varies per request, neither fixed choice is efficient: the win is matching each request to the cheapest model that can handle it.

Difficulty is per-request: Request difficulty follows a distribution — a fat head of easy queries and a tail of hard ones. A single model can't be optimal for both ends. Routing exploits the distribution: cheap capacity for the head, expensive capacity only for the tail.

Step 2 · A decision in front of the models

The router

Put a router between clients and a fleet of models of different cost/capability. Its one job is a decision — but on what basis, and what does it optimize?

The router takes each request and chooses which model to send it to, optimizing cost subject to a quality bar: use the cheapest model that will produce an acceptable answer. Behind it sits a portfolio of models — small/cheap, large/flagship, and task specialists, possibly across providers. The router hides this from clients (they call one endpoint) and turns "which model?" into an automatic, per-request optimization.

Cost under a quality constraint: Framing matters: the router minimizes cost while meeting a quality threshold, not just picks the cheapest blindly. That threshold — how much quality you'll trade for savings — is a business dial. The router is an optimizer over a model portfolio, per request.

Step 3 · Know the request

The routing signal

To route well, the router needs to estimate before answering how hard (or what type) each request is. Guessing wrong is costly. Where does that signal come from?

Design decision: How does the router estimate a request's difficulty/type before answering it?

The call: A fast, cheap classifier (heuristics, a small model, or embeddings) that scores difficulty/task. — A lightweight step — keyword/length heuristics, a small classifier model, or embedding-similarity to labeled examples — estimates the request's difficulty and/or task type quickly and cheaply. That score is the signal the routing decision is built on, and it must cost far less than the models it routes to.

Add a fast, cheap classifier that estimates each request's difficulty and/or task type before answering — via heuristics (length, keywords), a small classifier model, or embedding-similarity to labeled examples. Crucially it must cost far less than the models it routes to, or the routing overhead eats the savings. This score is the signal every routing decision stands on: get it right and you route cheaply and correctly; get it wrong and you misroute.

Cheap signal, big savings: The classifier is the router's eyes. It has to be much cheaper and faster than the target models (else you've added a tax), and reasonably accurate (else you misroute). Small models and embeddings make good routers precisely because they're cheap to run per request.

Step 4 · Cheap first, escalate

Tiered routing & the cascade

With a difficulty signal, you route easy→cheap and hard→flagship. But the classifier isn't perfect, and some "medium" requests are ambiguous. How do you route them without either overpaying or under-serving?

Use tiered routing plus a cascade: send a request to the cheapest model that the classifier thinks can handle it, and — for ambiguous cases — try cheap first, then escalate to a stronger model if the cheap answer is low-confidence or fails a check. The cascade catches classifier mistakes: most requests finish cheaply, only the ones that truly need it pay for the flagship. Balance the escalation threshold so you don't escalate everything (cost) or nothing (quality).

Tiers + cascade: Tiers: cheap / flagship / specialist. Cascade: attempt cheap, verify (self-consistency, a confidence score, or a validator), escalate on failure. This makes routing robust to an imperfect classifier — the cascade is a safety net that recovers quality when the cheap tier guesses wrong, while keeping average cost low.

Step 5 · The right specialist

Semantic / task routing

Difficulty isn't the only axis. A coding request is best served by a code model; a medical question by a domain fine-tune; a translation by a multilingual model. How do you route by what the task is, not just how hard it is?

Add semantic / task routing: use the classifier (or embeddings) to detect the request's domain/task and route to a specialist model that beats a general model on that slice — a code model for code, a fine-tuned model for your domain, a cheap multilingual model for translation. A specialist can be both cheaper and better on its niche than a general flagship, so task routing improves cost and quality simultaneously where a specialist exists.

Route by task, not just difficulty: Two routing axes: how hard (tier) and what kind (specialist). A tuned or purpose-built model often dominates a general one on its task at lower cost, so detecting task type and routing to the right specialist is a pure win — the router picks not just a size but a fit.

Step 6 · Serve it robustly

Multi-provider, fallback & caching

In production the router fronts many models across providers, any of which can be slow, rate-limited, or down — and many requests repeat. How does the router stay fast, cheap and reliable?

Make the router a resilient gateway over a model portfolio: cache repeated/identical prompts (the cheapest route — no model call at all), and add fallback so if the chosen model/provider is down or rate-limited, it fails over to an equivalent alternate (another provider or tier). Track per-model cost, latency and health to inform routing, and support multi-provider so you're not locked in and can route to whoever is cheapest/fastest for a given request. Routing + caching + fallback together cut cost and raise reliability.

Router as smart gateway: The router naturally becomes a gateway: caching for repeats, health-aware fallback across providers, and cost/latency accounting. This composes with (or is part of) an LLM gateway's auth/rate-limit/observability — routing is the decision layer on top of robust multi-model serving.

Step 7 · Get smarter over time

Learn routing from outcomes

A static router is only as good as its initial classifier — and model prices, capabilities and your traffic all change. How does routing stay optimal?

Close a feedback loop: log each request's route, cost, and an outcome signal (was the cheap answer good enough? did it escalate? user thumbs / eval score), and use it to recalibrate the classifier and the escalation thresholds — learning which requests the cheap model actually handles well and which truly need the flagship. Re-tune as models and prices change (a new cheap model may now cover cases that used to escalate). The router improves as it observes what routing decisions actually worked.

Calibrate on real outcomes: Routing is a prediction ("will the cheap model suffice?"), so it benefits from feedback like any predictor. Logging outcomes lets you calibrate thresholds to your real quality bar and adapt to a shifting model landscape — turning a hand-set router into a data-tuned one.

Step 8 · The sharp edges

Misrouting, latency & over-optimization

Routing has real risks: misrouting a hard query to a weak model (bad answer), the routing step's own latency/cost, keeping the classifier calibrated, and over-optimizing cost at the expense of quality.

Bound misrouting damage with the cascade (escalate on low confidence) and a conservative bias on high-stakes requests (when unsure, route up). Keep the routing overhead tiny — the classifier must be far cheaper/faster than the models, or it negates the savings (and adds latency to every request). Continuously calibrate against a real quality metric so the cost/quality trade stays where you want it. And don't over-optimize cost: define the quality bar first, then minimize cost under it — a router that saves money by quietly degrading answers is a failure, not a win.

Design for the unhappy path: Misroute → cascade + route-up-when-unsure. Routing tax → keep the classifier cheap. Drift → calibrate to a quality metric. Over-optimization → quality bar first, cost second. The router's value is real savings at the same quality; the failure mode is saving money by getting answers wrong.

You did it

You just designed an LLM router.

• O — n
• A —
• A —
• T — i
• S — e
• S — e
• L — e