What is model routing?
Model routing is the mechanism that decides which specific model handles a given request before any inference actually happens, classifying the request based on some observable signal, its content, its apparent complexity, its category, and directing it to the model best suited to handle it. It’s the upfront counterpart to the sequential, try-then-escalate approach covered in this collection’s discussion of model cascading, where routing makes its decision once, before any model has seen the request at all, while cascading observes an actual result from a cheaper model before deciding whether to escalate.
Why the router itself is a real piece of engineering, not a minor implementation detail
It’s easy to treat model routing as a small, incidental piece of a larger system, a quick check that sends a request to the right place before the “real” work of inference begins, but the router’s own accuracy directly determines whether the rest of the system’s cost and quality benefits actually materialize. A router that misclassifies requests sends easy traffic to an expensive model, wasting the cost savings the whole strategy was meant to capture, or sends hard traffic to a cheap model, degrading quality on exactly the requests where it matters most. The router isn’t a peripheral detail sitting outside the system’s real logic, it’s a component whose own correctness is just as consequential as the models it’s routing between.
This means a router deserves the same evaluation discipline given to any other model-driven component in a system, covered throughout this collection’s discussion of AI native testing, measuring how often it correctly classifies requests against a representative sample of actual traffic, rather than assuming a router works correctly simply because it was built using a reasonable-sounding approach.
The different mechanisms a router can use to make its decision
A router can be built several different ways, each with real tradeoffs in accuracy, cost, and latency. A rule-based router applies explicit, hand-written logic, keyword matching, category tags, simple heuristics, to decide where a request goes, which is fast, cheap, and fully predictable but only works well for requests that sort cleanly along whatever rules were written, missing the more ambiguous cases that don’t fit neatly into a predefined rule. A classifier-based router uses a small, dedicated model trained specifically to predict which downstream model a request should go to, which can capture more nuanced patterns than hand-written rules while still being fast and cheap to run relative to the models it’s routing between.
A third approach uses the language model itself, often a small one, to directly judge a request’s difficulty or category and output a routing decision, which can be more flexible and easier to build than training a dedicated classifier but adds its own inference cost and latency to every single request, a cost that has to be weighed against the savings the routing decision itself is meant to produce. Choosing between these approaches isn’t about finding the universally best one, it’s about matching the mechanism’s own cost and accuracy profile to what a specific deployment actually needs.
Why routing accuracy and routing cost pull against each other
A more sophisticated router, one using a trained classifier or a small language model to make its decision, tends to produce more accurate routing than a simple rule-based one, but that sophistication comes with its own computational cost and latency, added to every single request regardless of which downstream model ultimately handles it. This creates a real tension: a router expensive enough to route perfectly could end up consuming a meaningful share of the savings the whole routing strategy was supposed to deliver, while a router cheap enough to add negligible overhead may not be accurate enough to route reliably.
Finding the right balance here depends on the relative cost gap between the models being routed between, if the smaller model is dramatically cheaper than the larger one, a router that costs a bit more to get right is still clearly worth it, since even a meaningfully expensive router is small relative to the savings from correctly routing traffic to the cheaper model. If the cost gap between models is more modest, a cheaper, less sophisticated router often makes more practical sense, since there’s simply less savings available to justify investing heavily in routing accuracy.
What actually happens when routing gets it wrong
A router’s mistakes fall into two categories with meaningfully different consequences. Sending an easy request to an expensive model is a cost mistake, the request still gets answered correctly, just at a higher cost than necessary, an inefficiency rather than a quality failure. Sending a hard request to a model that can’t handle it well is a quality mistake, producing a response that may be confidently wrong or noticeably worse than what the correct routing decision would have produced, a failure that directly affects what the user experiences rather than just the deployment’s underlying economics.
Because these two kinds of mistakes carry such different consequences, a well-designed router is typically tuned asymmetrically, accepting more cost mistakes, routing borderline requests to the more capable model when uncertain, in exchange for fewer quality mistakes, mirroring the same conservative calibration philosophy covered in this collection’s discussion of semantic caching and model cascading, where a system’s failure mode matters as much as its overall accuracy rate.
How a router should handle genuinely ambiguous requests
Not every request sorts cleanly into an obvious category, and a router’s behavior on genuinely ambiguous cases, ones that could reasonably go either way, matters as much as its behavior on clear-cut ones. A router built without any explicit handling for ambiguity tends to make an arbitrary, effectively random decision on these borderline cases, which is a missed opportunity, since a router that recognizes its own uncertainty on a specific request can make a deliberate, informed choice, defaulting to the more capable model when genuinely unsure, rather than guessing with the same confidence it applies to clear-cut cases.
Building this kind of uncertainty awareness into a router is a meaningfully different engineering task than simply improving its average accuracy, it requires the router to have some calibrated sense of its own confidence, not just a raw classification decision, and investing in this capability tends to matter more for overall system quality than chasing marginal improvements in accuracy on the cases the router was already confident about and handling correctly.
Why a router needs its own monitoring separate from the models it routes to
A production system’s overall quality metrics can look healthy even while its router is performing poorly, if the downstream models being routed to are individually strong enough to compensate for occasional misrouting, which means monitoring only the system’s end-to-end quality can miss a router that’s quietly making worse decisions than it should be. Tracking the router’s own accuracy specifically, how often it sends a request to the model that would have produced the best result, gives visibility into a failure mode that end-to-end metrics alone can obscure.
This distinction matters because a router’s accuracy can degrade gradually as the actual distribution of incoming requests shifts away from whatever it was originally built and validated against, and without dedicated monitoring specific to the routing decision itself, that gradual degradation can go unnoticed for a long time, quietly eroding the cost savings or quality benefits the routing strategy was originally built to deliver.
Common mistakes teams make around model routing
1. Treating the router as a minor implementation detail rather than a component whose own accuracy directly determines whether the system’s cost and quality goals are actually met.
2. Choosing a routing mechanism’s sophistication without weighing its own added cost and latency against the savings it’s actually meant to produce.
3. Tuning a router for overall accuracy without accounting for the asymmetric consequences of cost mistakes versus quality mistakes.
4. Building a router with no explicit handling for genuinely ambiguous requests, letting it guess with unwarranted confidence on cases it should recognize as uncertain.
5. Monitoring only end-to-end system quality, missing gradual router degradation that downstream model strength happens to be compensating for, at least for now.
What connects these mistakes is underestimating how much of a routing-based system’s actual performance traces back to the router’s own quality, not just the quality of the models it routes between, a router built and evaluated with the same rigor given to the rest of the system is what actually delivers the cost and quality benefits routing is meant to provide, rather than quietly undermining them through decisions nobody’s specifically watching.
The deeper point about model routing is that it moves the hard problem of matching capability to need from something decided inside a single model’s response to something decided deliberately, upfront, by a system a team actually designs and can inspect, and treating that upfront decision with the same care given to model selection itself is what separates a routing strategy that reliably delivers on its promise from one that merely looks efficient on paper.