How Model Failover Works in idapt

A model in our catalog is often servable by several providers at different prices and reliability levels, and providers degrade without warning. Routing is the layer that turns that mess into one promise: you name a model, you get an answer. Here is how it works.

Cheapest-first, with receipts

When a request names a model, the router orders that model's healthy providers by effective cost and tries the cheapest first. If a provider fails or degrades mid-request, the router fails over to the next, transparently: the retry is ours to handle, not yours to notice.

Two properties keep this checkable:

• Attribution: every reply records which provider served it. Routing decisions are inspectable, never mystical.
• Health is measured, not assumed: real success rates and latency feed the routing table, so a flaky provider loses traffic before you would have noticed it flaking.

Sticky pinning: consistency beats pennies

Failover has a subtle enemy: mid-conversation provider hops. Two providers serving "the same" model can differ slightly (sampling defaults, context handling), and a conversation that silently switches feels haunted. So routing is sticky: once a conversation lands on a provider, it stays there while that provider stays healthy. Failover is for failures, not for chasing a fractionally cheaper token mid-thread.

Media models fail differently

Text models get cross-provider failover. Image, video, and speech models usually map to a single provider, so their failure handling is bounded same-provider retry, and (for video, which runs asynchronously) failed generations refund automatically rather than billing for nothing.

The layers above

Failover composes with the rest of routing: your provider preferences and blocks constrain candidates, BYOK keys route matching traffic to your accounts, and local models are a provider lane with a very short cable. The auto-router chooses models; this layer chooses how a model's request survives the real world.

The upshot: multi-provider reliability engineering, delivered as the absence of a certain kind of bad afternoon.