The whole design, in writing
Learn AI system design by building an LLM router that sends each request to the cheapest model that can handle it step by step. An interactive guide covering why one-model-for-all is wasteful, classifying query difficulty, tiered routing with a cheap-first cascade, semantic/task routing to specialized models, multi-provider serving with fallback and caching, learning routing from outcomes, and the unhappy paths (misrouting, routing latency, calibration).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What does an LLM router do?
Not every request needs your biggest, most expensive model. "Reformat this JSON" and "reason step-by-step through this legal argument" cost the same if you send both to the flagship — but the first could be handled by a model 10–50× cheaper and faster with identical quality. Send everything to the big model and you overpay massively; send everything to the small one and hard queries fail.
An LLM router picks, per request, the cheapest model that can still get it right. It classifies the request's difficulty/type, routes easy ones to a small model and hard ones to the flagship (or a specialist), and escalates when the cheap model isn't confident. Same quality, a fraction of the cost — the routing decision is the whole system.
What the new pieces do
- Clientclient
- A stream of requests of wildly varying difficulty — from "reformat this" to "reason through this proof". Most are easy.
Step 1 · One model for all is wasteful
The cost/quality bind
The two naive options: route everything to the big model, or everything to the small one. Why is each a bad deal?
Send all requests to one model. Big model for all, or small for all — what's wrong with each?
Most requests are easy and don't need a flagship, so big-for-all burns money and latency on trivial work; but small-for-all gives wrong/weak answers on the genuinely hard requests. Since difficulty varies per request, the efficient answer is to match each request to the cheapest model that can handle it.
They're not equivalent trade-offs to ignore — big-for-all can be 10–50× more expensive than necessary on easy queries, and small-for-all degrades quality where it matters. Per-request routing captures the savings without the quality loss.
The big model is more capable but far more expensive and slower, and that capability is wasted on the easy majority. Matching model to request difficulty saves most of the cost at the same quality.
Big-for-all overpays — most requests are easy and don't need flagship capability, so you burn 10–50× the necessary cost and latency on trivial work. Small-for-all fails the hard minority with weak or wrong answers. Because difficulty varies per request, neither fixed choice is efficient: the win is matching each request to the cheapest model that can handle it.
Why this piece earns its place
The distribution argument is true in general and unproven for your traffic, which makes the first deliverable a histogram rather than a router. Sample a week of real prompts, hand-label a few hundred by whether the cheap model's answer would have been acceptable, and you have a ceiling on what routing can save before anyone writes routing code. That number decides the project: a product where nearly every request is a long tool-calling chain has almost no easy head, and a router there is added latency plus one more thing to page someone about. It also decides how you pitch it. The savings land on a bill that nobody using the product ever sees, while the same swap shortens the wait on every request it touches — and the wait is what gets the work funded, because it gives someone outside finance a reason to want this. One warning about the accounting: the price of a request is not the price of the call. A weak answer the user rewords and resends costs two calls plus their patience, so cost per resolved request is the honest denominator.
Step 2 · A decision in front of the models
The router
Put a router between clients and a fleet of models of different cost/capability. Its one job is a decision — but on what basis, and what does it optimize?
The router takes each request and chooses which model to send it to, optimizing cost subject to a quality bar: use the cheapest model that will produce an acceptable answer. Behind it sits a portfolio of models — small/cheap, large/flagship, and task specialists, possibly across providers. The router hides this from clients (they call one endpoint) and turns "which model?" into an automatic, per-request optimization.
Why this piece earns its place
The single endpoint is cheap to draw and expensive to honour, because models are not drop-in replacements for one another: system prompts land differently, JSON and tool-call support varies, context windows differ, and a prompt tuned against the flagship can come back malformed from the cheap model. If per-model prompt adaptation lives in the client, every client reimplements it and the portfolio freezes — nobody dares add a model. Keep that adaptation inside the router and it stays a portfolio you can change. Two consequences worth saying out loud. The response has to name the model that answered, in metadata rather than in the body, or a quality complaint is unattributable and per-model cost accounting is guesswork. And routing needs a per-request override — a model pin for evals, reproducible runs, or a flow that is contractually required to name its model — while the default stays automatic; that knob belongs in the request, and the policy mapping difficulty to tiers belongs in config. The long payoff: once no client names a model, you can retire a provider without a client migration.
- portfoliocheap · flagship · specialist
- optimizecost under quality bar
- one endpointclients unaware
What the new pieces do
- Routergateway
- Decides, per request, the cheapest model that can handle it — then forwards, escalating if the cheap one falls short.
Step 3 · Know the request
The routing signal
To route well, the router needs to estimate before answering how hard (or what type) each request is. Guessing wrong is costly. Where does that signal come from?
How does the router estimate a request's difficulty/type before answering it?
Calling the expensive model to decide whether to call the expensive model defeats the purpose. The signal must come from something much cheaper than the models you're routing to.
Random routing sends hard queries to weak models and easy ones to the flagship — the worst of both. You need an actual difficulty/type estimate.
A lightweight step — keyword/length heuristics, a small classifier model, or embedding-similarity to labeled examples — estimates the request's difficulty and/or task type quickly and cheaply. That score is the signal the routing decision is built on, and it must cost far less than the models it routes to.
Add a fast, cheap classifier that estimates each request's difficulty and/or task type before answering — via heuristics (length, keywords), a small classifier model, or embedding-similarity to labeled examples. Crucially it must cost far less than the models it routes to, or the routing overhead eats the savings. This score is the signal every routing decision stands on: get it right and you route cheaply and correctly; get it wrong and you misroute.
Why this piece earns its place
Most of the accuracy here comes from what you hand the classifier, not from which classifier you pick. The unit is the whole request, not the last user message: a later turn carrying tool results and a long system prompt is not the difficulty of the sentence at the end of it. The cheapest signals sit outside the text entirely — how many turns deep this is, which tools are attached, whether the caller demanded a strict output schema — and they separate easy from hard about as well as the prose does, for nothing. Where the classifier runs matters as much as what it is. Put it behind a network call and every request pays a round trip before any work starts, and its tail latency lands on every request's tail rather than its median — so a blunter classifier with constant cost beats a sharper one that occasionally stalls. A small classifier also has a small context window, so long requests get scored on a prefix: the window forces that, not the bill. Treat having to truncate a request in order to classify it as a difficulty signal in its own right.
- classifierdifficulty + task
- cheap« target models
- signalthe routing basis
What the new pieces do
- Classifierservice
- A fast, cheap step that estimates the request's difficulty/type — the signal the routing decision is built on.
Step 4 · Cheap first, escalate
Tiered routing & the cascade
With a difficulty signal, you route easy→cheap and hard→flagship. But the classifier isn't perfect, and some "medium" requests are ambiguous. How do you route them without either overpaying or under-serving?
Use tiered routing plus a cascade: send a request to the cheapest model that the classifier thinks can handle it, and — for ambiguous cases — try cheap first, then escalate to a stronger model if the cheap answer is low-confidence or fails a check. The cascade catches classifier mistakes: most requests finish cheaply, only the ones that truly need it pay for the flagship. Balance the escalation threshold so you don't escalate everything (cost) or nothing (quality).
Why this piece earns its place
The cascade's real cost is not the second call, it is serial latency landing on the requests that were already hardest: an escalated request waits out the cheap model, then the check, then the strong model. Budget for that in the tail, and let a caller cap it — a request that cannot afford two round trips should say so and go straight to the strong tier. It also collides with streaming. You cannot unsend tokens, so a cheap answer already streaming cannot be withdrawn when the check fails. Either you buffer the cheap answer until the check passes, which pushes time-to-first-token out to the cheap model's full completion time on every request in order to protect the minority that will fail it, or you do not cascade on streaming endpoints and route them on the classifier alone. Deciding that per endpoint is the honest answer. Escalation rate is worth watching for a reason beyond cost, too: it moves when traffic shifts, and it moves when a provider silently repoints an alias at a new build of the cheap model — often the only warning you get that the model underneath you changed.
- tierscheap → flagship
- cascadetry cheap, escalate
- thresholdtune escalate rate
What the new pieces do
- Cheap Modelservice
- Handles the easy majority at a fraction of the cost/latency of the flagship model.
- Strong Modelmint
- The expensive, capable model reserved for the genuinely hard requests (and escalations from the cheap tier).
Step 5 · The right specialist
Semantic / task routing
Difficulty isn't the only axis. A coding request is best served by a code model; a medical question by a domain fine-tune; a translation by a multilingual model. How do you route by what the task is, not just how hard it is?
Add semantic / task routing: use the classifier (or embeddings) to detect the request's domain/task and route to a specialist model that beats a general model on that slice — a code model for code, a fine-tuned model for your domain, a cheap multilingual model for translation. A specialist can be both cheaper and better on its niche than a general flagship, so task routing improves cost and quality simultaneously where a specialist exists.
Why this piece earns its place
Cheaper and better is a claim about one request, and on its slice it holds. What it does not price is the standing cost of keeping that specialist in the portfolio. Each one is its own prompt template, its own output quirks, its own eval set, its own deprecation notice to absorb. Two are a clean win; eight are eight prompt suites drifting at different rates, and the router's quality becomes the minimum across all of them. Add one when a slice is large enough that its savings pay for that maintenance, not because a good model exists. Where the specialist runs changes the arithmetic as well. A hosted code or translation model bills per token like everything else on this page, so it costs nothing on the days nobody asks for it. A domain fine-tune you host yourself is rented capacity instead — an idle replica costs the same at three in the morning as at peak — so that kind of specialist only pays above a traffic floor for its slice, and that floor is a number worth asking for before you commit. And decide the ambiguous case deliberately: a request that is half code and half prose has no clear owner, so when task detection is not confident, fall back to the general tier rather than the nearest specialist.
- task detectcode/medical/lang
- specialistbeats general on niche
- cheaper + betteron its slice
What the new pieces do
- Specialistservice
- A model tuned for a specific task/domain (code, medical, a fine-tune) that beats a general model on that slice.
Step 6 · Serve it robustly
Multi-provider, fallback & caching
In production the router fronts many models across providers, any of which can be slow, rate-limited, or down — and many requests repeat. How does the router stay fast, cheap and reliable?
The chosen cheap model's provider is rate-limiting you mid-traffic spike. What should the router do?
Queuing adds unbounded latency to a request that could be served right now by an equivalent model elsewhere — the user pays for a provider-side problem they had nothing to do with.
The router already tracks which models are interchangeable for a given tier/task, so a rate-limited or down provider is just one option going temporarily unavailable — route to the next-best equivalent and keep serving.
Pushing the failure back to the caller defeats the point of a router that is supposed to hide model/provider details behind one endpoint — resilience belongs in the routing layer, not in every client.
Make the router a resilient gateway over a model portfolio: cache repeated/identical prompts (the cheapest route — no model call at all), and add fallback so if the chosen model/provider is down or rate-limited, it fails over to an equivalent alternate (another provider or tier). Track per-model cost, latency and health to inform routing, and support multi-provider so you're not locked in and can route to whoever is cheapest/fastest for a given request. Routing + caching + fallback together cut cost and raise reliability.
Why this piece earns its place
Failover is the one routing decision made under pressure, and it quietly bypasses the quality bar everything else is held to. "These two are interchangeable for this tier" is a hand-written claim that rots as both sides ship new versions, so every equivalence pair needs its own eval — otherwise an outage becomes a quality incident that nobody records as one, because the dashboard shows requests served. Mechanically, health tracking has to be a circuit breaker with a half-open probe, not a retry rule. Without the breaker, every request during an outage pays the full timeout before failing over, so a degraded provider costs latency across all traffic instead of losing you one tier; with it, a single probe pays that cost and everything else routes straight past. The cache needs a flag of its own. How repetitive your traffic is sets the ceiling on hit rate; your prompt template decides whether you get to collect any of it. A template that interpolates a user id, a timestamp or a rolling session summary never produces an identical prompt twice, so it zeroes out whatever repetition the traffic had. Confirm the cached text can repeat before tuning anything else about the cache.
- cacheno-call route
- fallbackprovider/tier failover
- multi-providerno lock-in
What the new pieces do
- Cachecache
- Serves repeated/identical prompts without calling any model — the cheapest route of all.
Step 7 · Get smarter over time
Learn routing from outcomes
A static router is only as good as its initial classifier — and model prices, capabilities and your traffic all change. How does routing stay optimal?
Close a feedback loop: log each request's route, cost, and an outcome signal (was the cheap answer good enough? did it escalate? user thumbs / eval score), and use it to recalibrate the classifier and the escalation thresholds — learning which requests the cheap model actually handles well and which truly need the flagship. Re-tune as models and prices change (a new cheap model may now cover cases that used to escalate). The router improves as it observes what routing decisions actually worked.
Why this piece earns its place
The feedback this loop collects is censored, and that is the flaw to name out loud. You only ever observe the outcome of the model you actually picked, so the log tells you how often escalation was needed but never which flagship-routed requests the cheap model could have handled. Train on it as it stands and the classifier's frontier freezes wherever day one put it, growing more confident about a boundary it no longer tests. The fix is deliberate exploration: route a small slice against policy, or — cheaper and safer — shadow the cheap model offline on flagship traffic and compare the two answers after the fact. That work is off the request path, so it batches and runs on spare capacity. Version the policy alongside every logged decision, because routing shapes the data it is later trained on: a threshold edit shifts the whole distribution of what you observe, and without a policy version in the row you cannot separate a model regression from a routing change you made last Tuesday.
- logroute + cost + outcome
- calibrateclassifier + thresholds
- adaptto new models/prices
What the new pieces do
- Feedbackservice
- Learns from outcomes (was the cheap answer good enough?) to recalibrate the classifier and routing thresholds.
Step 8 · The sharp edges
Misrouting, latency & over-optimization
Routing has real risks: misrouting a hard query to a weak model (bad answer), the routing step's own latency/cost, keeping the classifier calibrated, and over-optimizing cost at the expense of quality.
Bound misrouting damage with the cascade (escalate on low confidence) and a conservative bias on high-stakes requests (when unsure, route up). Keep the routing overhead tiny — the classifier must be far cheaper/faster than the models, or it negates the savings (and adds latency to every request). Continuously calibrate against a real quality metric so the cost/quality trade stays where you want it. And don't over-optimize cost: define the quality bar first, then minimize cost under it — a router that saves money by quietly degrading answers is a failure, not a win.
Why this piece earns its place
The two failures here do not surface the same way, and that asymmetry is the whole problem. Cost arrives as a number, on a schedule, in front of somebody whose job is to read it. Quality arrives as an anecdote in a support queue, months later, attached to no particular release. A router drifts in the direction only one of those two can see. Two habits make the other one visible. First, stop relying on route up when unsure to protect the requests that matter: it fires only when the classifier knows it is unsure, and the misroutes that end in a complaint are the confident ones. Name the surfaces where a cheap answer is never acceptable and pin them to the strong tier in config, so protection is a list someone can read rather than a judgement made per request. Second, measure savings against a fixed basket of replayed requests, not against last month's bill — cost per request also falls when traffic gets easier, and a router credited for a seasonal shift will be blamed for the next one. That basket is also how you price a threshold edit before it ships, which matters because a threshold is one line of config that moves the quality of all traffic the moment it lands.
You did it
You just designed an LLM router.
Everything you assembled, in order
- One-model-for-all is a bind: big overpays on the easy majority, small fails the hard minority.
- A router picks the cheapest model that meets a quality bar, per request, over a model portfolio.
- A fast, cheap classifier estimates difficulty/task — the signal every routing decision stands on.
- Tiered routing + a cheap-first cascade (escalate on low confidence) make routing robust to classifier errors.
- Semantic/task routing to specialists is cheaper AND better on their niche — a pure win.
- Serve robustly: cache repeats, fail over across providers, and track cost/latency/health.
- Learn routing from outcomes; keep the quality bar first and minimize cost under it (don't degrade quality).