The Sorting Hat spends about four seconds on most students. A glance inside, a decision, next kid.
Then it gets to Harry and stalls. This one is hard. Plenty of courage, a good mind, a complicated history. The hat takes its time, weighs the options, argues with the student. Difficult cases get the full deliberation. Easy cases get four seconds.
Nobody at Hogwarts thinks this is strange, because it would be absurd to give every eleven-year-old a twenty-minute psychological evaluation when most of them are obviously a Hufflepuff at first glance.
Most AI systems in production do the absurd thing. Every request, no matter how trivial, goes to the biggest, most expensive model the team has access to. The four-second cases and the Harry cases all get the full deliberation, at full price.
The Cost Gap Is Not Small
The price difference between model tiers is one of the largest arbitrage opportunities in your stack. A frontier model typically costs 15-60x more per token than the small model from the same provider. Not 15-60 percent. Fifteen to sixty times.
That gap would be irrelevant if small models were bad. They are not. On mechanical tasks (classify this ticket, extract these fields, reformat this text, decide which of four categories this belongs to) current small models perform at or near frontier level. The frontier premium buys deep reasoning, long-horizon planning, and nuanced synthesis. It buys nothing on tasks that do not need those things.
Now look at real traffic. In most production systems, the mechanical tasks are the bulk of the volume. Routing decisions, tagging, extraction, yes/no gates, short factual lookups. The Harry cases (genuinely hard reasoning) are a minority of requests.
Which means the default setup, one frontier model for everything, spends most of its budget buying reasoning that is never used.
Four Ways to Route
A router is anything that decides which model handles a request. It can be one line of code or a model in its own right.
| Pattern | How It Decides | Best For |
|---|---|---|
| Static routing | Task type is mapped to a model at design time | Systems with distinct, predictable task types |
| Classifier routing | A cheap model or heuristic grades each request's difficulty at runtime | Open-ended traffic like chat or support |
| Cascade | Cheap model tries first; escalate on failure or low confidence | Tasks with a checkable success condition |
| Pipeline routing | Each stage of a workflow gets its own model | Multi-step agent systems |
Static routing is the simplest and most underrated. If your system has a "summarize this document" feature and a "tag this record" feature, those are different endpoints in your code already. Assigning them different models is a config change, not an architecture project.
Classifier routing puts a small model at the front door. It reads the request and decides: easy or hard? The classifier call costs a fraction of a cent, and it only needs to be right often enough that the savings on correctly-routed easy traffic outweigh the occasional misroute. It usually is.
Cascades flip the order. Try cheap first, check the result, escalate if it fails. This works when failure is detectable: the output fails schema validation, a confidence score comes back low, a test fails. The catch is latency. Failed attempts add time, so cascades fit background jobs better than live chat.
Pipeline routing is where multi-agent systems earn their keep. Our own platform, JESTR, runs this way: a small, fast model reads each incoming question and routes it, parallel workers fetch data, and a mid-tier model synthesizes the final answer. The expensive model appears at the one stage that needs judgment. The routing stage runs hundreds of times a day on a model that costs pennies, and no user has ever noticed the difference. That is the point.
When Not to Bother
Routing is an optimization, and optimizations have prerequisites.
If your volume is small, skip it. A prototype doing 50 requests a day does not have a cost problem worth an engineering sprint. Build on the best model, learn what your traffic looks like, route later.
If your workload is uniformly hard, skip it. A system that only does complex legal analysis has no easy bucket to move down a tier.
And if you cannot evaluate output quality yet, definitely skip it. Routing without evals is how you trade visible cost for invisible quality loss. You will not know the small model is quietly worse until a customer tells you.
How to Start
Do not build a router first. Look at your logs first.
Bucket a week of requests by task type. In most systems, two or three buckets dominate the volume, and at least one of them is obviously mechanical.
Take that bucket and run it through a model one tier down, in parallel with production, and compare the outputs. Not on vibes. On an eval, even a rough one: exact-match for classification, field accuracy for extraction, a rubric scored by a judge model for anything fuzzier.
If the small model holds up, switch the bucket and keep the escalation path. If it does not, you spent a day learning something true about your workload, which is more than most teams know about theirs.
Then repeat, one bucket at a time. The teams that do this typically end up with frontier models handling a fraction of their traffic and a bill that dropped by more than half. The hat spends four seconds on the easy ones. Save the deliberation for Harry.
