What is model selection?

Summary slides

Why defaulting to the most capable available model is a common but costly mistake

The most intuitive approach to model selection is reaching for the single most capable model available, reasoning that more capability can only help and never hurt, but this intuition misses that capability comes bundled with real cost and real latency, and a considerable share of tasks don’t actually need the extra capability a larger, more expensive model provides. A smaller, faster, cheaper model that handles a task just as well as a larger one is the better choice for that task, and paying the premium for extra capability that never gets used is a real, ongoing cost with no corresponding benefit.

This mistake is easy to make because the downside isn’t immediately visible, a team using an unnecessarily large model for a simple task doesn’t see an obvious failure, everything works, the responses are good, and the excess cost and latency simply become an invisible tax baked into the product’s ongoing economics rather than a problem that surfaces clearly enough to prompt a second look. Recognizing this requires deliberately asking whether a smaller model would perform the task just as well, rather than assuming the largest available option is automatically the safest choice.

The three dimensions that drive a selection decision

Model selection ultimately comes down to weighing three dimensions against each other for a specific task: how capable a model needs to be to handle that task well, how much latency the application can tolerate before a slow response starts hurting the experience, and how much cost the deployment can sustain given how often the model will actually be called. A task requiring nuanced, open-ended reasoning weighs capability heavily and may tolerate a considerably higher cost and latency in exchange for getting that reasoning right. A task running at very high volume, classifying incoming support tickets, extracting fields from a large batch of documents, weighs cost and speed far more heavily, since even a small per-call cost or delay compounds dramatically at that kind of scale.

No single model wins across all three dimensions simultaneously, which is exactly why model selection is a genuine tradeoff rather than a search for one universally correct answer, the right choice depends entirely on which of these three dimensions matters most for the task at hand, and a team that treats one model as the default answer for every task within a product is very likely leaving real efficiency on the table somewhere.

Why evaluating a model against the actual task beats trusting a generic benchmark

Public benchmark scores offer a convenient way to compare models at a glance, but a benchmark score measures performance against a fixed, generic set of test cases that may not resemble the specific task a team is trying to solve, and a model that scores well on a broad, general benchmark can still perform noticeably worse than a competitor on the narrower kind of request a particular application actually handles. This connects directly to the broader discussion of AI native testing covered elsewhere in this collection, the only evaluation that reliably predicts how a model will perform in production is one built around a representative sample of the actual workload that model will face.

A team that selects a model purely based on its ranking on a popular public benchmark, without validating that ranking against its own specific use case, is trusting a proxy measurement instead of the real one, and the gap between those two can be considerable, especially for tasks involving domain knowledge, particular formatting requirements, or edge cases a generic benchmark was never designed to capture in the first place.

Why a single product often benefits from more than one model

Because the right tradeoff between capability, latency, and cost varies by task, a single product frequently benefits from using different models for different parts of its functionality rather than committing to one model across the board, connecting directly to the broader discussion of mixed-model serving strategies covered elsewhere in this collection. A customer support product might use a smaller, faster model to handle routine, well-defined questions while routing genuinely complex or ambiguous requests to a larger, more capable model, capturing the cost and speed benefits of the smaller model for the bulk of routine traffic while still reserving the larger model’s capability for the cases that genuinely need it.

Designing a system this way requires the same routing logic covered in the discussion of mixed-model serving, correctly identifying which requests actually need the more capable model rather than sending everything to the same place, but the payoff, meaningfully lower average cost and latency without sacrificing quality on the requests that matter most, is often considerable enough to justify that added routing complexity.

Why model selection isn’t a one-time decision

The pace at which new models get released, each with its own capability, cost, and speed profile, means a model selection decision made even a few months ago deserves periodic reassessment rather than being treated as permanently settled. A newly released model might offer the same capability at meaningfully lower cost, or meaningfully better capability at the same cost, and a team that never revisits its original choice misses out on exactly the kind of efficiency gain that’s readily available simply by staying current with what’s actually on offer.

This doesn’t mean chasing every new release reflexively, switching models has real switching costs of its own, re-validating behavior, re-testing edge cases, potentially adjusting prompts tuned for the previous model’s particular behavior, and those costs need to be weighed against the actual benefit a newer model offers for the specific task at hand. The right cadence is somewhere between never revisiting the decision and chasing every release, periodically checking whether a meaningfully better option has become available, informed by actual testing rather than by reputation or hype alone.

How switching costs shape what “better” actually means in practice

A model that scores better on paper isn’t automatically worth switching to, since prompts, formatting expectations, and downstream logic are often tuned, sometimes unintentionally, around a specific model’s particular quirks and tendencies, and switching to a different model can surface subtle behavioral differences that require real validation work to catch before they become a production problem. This is why a considered model selection decision accounts for the cost of switching itself, not just the isolated comparison of one model’s capabilities against another’s, and why a genuinely marginal improvement often isn’t worth the disruption of switching, while a substantial one usually is.

This dynamic favors building systems in a way that keeps switching costs as low as practical from the start, avoiding tight coupling to any one model’s own idiosyncrasies where possible, since a team that’s designed for this flexibility from the beginning has a much easier time actually capturing the benefit of a better model when one becomes available, rather than being effectively locked into an earlier choice by how much work switching would require.

Common mistakes teams make around model selection

1. Defaulting to the most capable available model for every task without evaluating whether a smaller, cheaper model would perform that specific task just as well.

2. Trusting a generic public benchmark ranking without validating it against the team’s own specific task and actual workload.

3. Committing to a single model across an entire product rather than considering whether different parts of that product would benefit from different models suited to their specific requirements.

4. Treating an initial model selection decision as permanently settled, missing meaningful cost or capability improvements that become available as new models are released.

5. Underestimating the real cost of switching models, comparing capability scores in isolation without accounting for the validation and adjustment work a switch would actually require.

What connects these mistakes is treating model selection as a single, static decision rather than an ongoing discipline that weighs capability, cost, and latency against a task’s actual requirements, revisited periodically as both the task’s needs and the available models themselves continue to evolve.

The deeper point about model selection is that a model’s raw capability is only one input into whether it’s actually the right choice for a given task, and a team that evaluates selection through the narrower lens of “which model is smartest” rather than the fuller question of “which model fits this specific task’s actual requirements” tends to end up paying for capability it doesn’t need in places that didn’t require it, while potentially underinvesting in capability where a task genuinely does call for it.