Model Routing and Provider Failover 
Not Betting the Product on One Model 

The first version of every AI feature hardcodes one model. That is the correct decision — an abstraction written before you know how the feature behaves in production is an abstraction over the wrong things. But the shortcut has a shelf life, and it usually expires in one of three ways: the provider has a bad afternoon and your feature is simply down, the model you built on is deprecated with a migration window measured in weeks, or you discover that eighty percent of your calls are doing work a model costing a fifth as much would do just as well. Each of those is survivable in isolation. What makes them expensive is discovering, at the moment one of them happens, that the model identity is threaded through forty files and there is no way to evaluate whether a replacement is actually as good.

The three risks, in the order they usually bite

Availability comes first because it is the most visible. Model providers have incidents, and the incidents that matter are usually not clean outages but elevated latency and elevated error rates that turn a two-second feature into a thirty-second one. If your feature is on a request path, that degradation is your degradation. Deprecation comes second and is more disruptive than teams expect: models are retired on the provider's schedule, and a migration you did not plan lands in the middle of a quarter with a hard date attached and no obvious way to confirm the new model behaves like the old one. Cost is third but grows fastest, because the model you chose during development is almost always the strongest one available — perfectly reasonable while proving the feature works, and clearly wasteful once you can see that most of your traffic is classification, extraction, or summarisation that a smaller model handles indistinguishably. A fourth risk is quieter and worth naming: quality changes under you. Providers update models behind a stable name, and a prompt tuned against one revision can behave measurably differently against the next, which you will only notice if you are measuring.

The abstraction that works, and the one that does not

The instinct is to build a universal adapter that normalises every provider's API into one interface, and it is usually a mistake — you end up maintaining a lowest-common-denominator surface that hides the provider-specific features you actually wanted, and it rots every time a provider ships something new. The useful abstraction is narrower and sits one level up: your application code should call a named capability rather than a model, so that the calling code asks for something like classify-support-ticket or draft-reply, and a small routing layer owns which model, which prompt version, and which parameters serve that capability today. The prompt travels with the model in that mapping, because a prompt tuned for one model is not automatically correct for another, and separating them is what makes model swaps quietly degrade quality. Keep provider SDKs at the edge rather than wrapping them, so you can use provider-specific capabilities where they matter, and keep the routing table as configuration you can change without a deploy. The test of whether you have drawn the boundary correctly is simple: switching the model behind one capability should be a config change plus an eval run, and should touch no application code.

Routing by task, not by preference

• Classification, extraction, tagging, and routing — high volume, narrow output, objectively checkable. These are where a small fast model usually matches a large one, and where most of your unnecessary spend lives.
• Summarisation and drafting where a human reviews the output. Mid-tier models are typically sufficient because the human is the quality gate; spend the savings on latency instead.
• Reasoning-heavy work — multi-step analysis, code generation, anything where a subtle error is expensive and unreviewed. Use the strongest model available and do not optimise here first.
• Anything on an interactive path where a user is watching a cursor blink. Latency is the dominant quality attribute; a faster model with a slightly lower score often produces a better product.
• Bulk and offline work with no user waiting. Route to whatever is cheapest that passes evals, and use batch endpoints where the provider offers them.

Failover that helps rather than hurts

Failover between model providers is not the same as failover between database replicas, and treating it that way produces a system that fails in a more confusing way than it would have alone. A fallback model will produce different output, in a different format, at a different quality level, and if your parsing is strict or your feature depends on structured output, the fallback can fail in ways the primary never did — which means the fallback path has to be tested, not merely configured. Start with the cheap correctness controls: an aggressive timeout so a degraded provider fails fast instead of holding your request threads, a circuit breaker that stops sending traffic to a provider that is clearly unwell instead of retrying into it, and retries with backoff and jitter for transient errors only. Then decide honestly, per capability, what degradation is acceptable. For an asynchronous feature the best fallback is usually to queue and retry later rather than to answer worse. For an interactive feature a smaller or alternative-provider model behind the same capability is worth having, provided you have run the evals to know what you are getting. And for some features the right answer is a clear, honest unavailable state — an AI feature that is transparently off for eleven minutes damages trust far less than one that quietly returns worse answers for a day.

Evals are what make any of this safe

Everything above depends on being able to answer one question: is this model, on this prompt, good enough for this task. Without that, routing decisions are guesses, failover is a gamble, and a forced migration becomes a month of anxiety. This is why an evaluation harness is the highest-leverage thing to build in an AI product and the thing most teams postpone. It does not need to be elaborate at the start: a few hundred representative inputs per capability, with expected outputs or a grading rubric, and a script that scores any model-and-prompt combination against them. Once that exists, swapping a model stops being a leap of faith and becomes an experiment with a number attached, cost optimisation becomes a series of cheap tests rather than a debate, and provider quality drift becomes something you detect rather than something a customer reports. Add a small production sample review — a handful of real outputs graded weekly — because offline evals drift from reality, and the gap between them is where most quality regressions hide.

The honest trade

None of this is free, and multi-model architecture has a real cost: more configuration, more prompt variants to maintain, more paths to test, and a genuine risk of building an elaborate routing system for a feature that has one model and forty users. The proportionate version scales with the stakes. If you have one AI feature in beta, hardcode the model, keep the model name in one place, and write down what you would do if the provider went down for an hour — that is enough. If the feature is on a paying customer's critical path, invest in the capability abstraction, the eval harness, and a tested fallback for the specific capabilities that cannot be unavailable. If AI is your product and inference is a material line in your cost structure, routing by task is where the money is and it is worth doing carefully. The mistake to avoid at every size is the one that is hardest to reverse: letting model identity and prompt text spread through the codebase so that any future change requires touching everything.

How Infiniti Tech Partners approaches this

We are usually brought into this after a forcing event — a deprecation notice, an inference bill that grew faster than usage, or an outage that made an AI feature's single point of failure impossible to ignore. The work starts with an inventory of every model call in the codebase, which frequently surprises the team, and a per-call judgement about what the task actually requires versus what it currently uses. From there we build the capability boundary and the routing configuration, port prompts across with their models rather than independently, and stand up the eval harness first because every subsequent decision depends on it. Where a feature genuinely cannot be unavailable we build and test the fallback path properly, including the failure modes the fallback introduces, and where it can we usually recommend queueing over degrading. Cost work tends to fund the rest of it: routing high-volume narrow tasks to smaller models typically removes a large fraction of spend without a measurable quality change, and now you have the evals to prove that rather than assert it. If you are staring at a migration deadline or an inference bill that has stopped making sense, this is a well-bounded engagement with a number at the end.