At that rate, a router based only on “use the cheapest model unless the request looks hard” is too vague. I need to know the task types, tenant contracts, allowed regions and providers, deadlines, and which mistakes have real consequences. Routing is a policy decision constrained by capacity, not just a model ranking. A pinned provider is a contract. Its traffic cannot silently go to another provider because one dashboard says the alternative looks similar.

I would classify only from information available before routing: customer pin, task type, declared tool capabilities, input length, data boundary, and required quality tier. For each allowed model, keep a versioned capability record, evaluation results by task slice, expected latency under current load, price, and quota headroom. Separate a model's theoretical quality from the current service's ability to finish within the deadline. A model with excellent offline accuracy but a 20 minute queue is not a usable choice for a live support response.

The router should reserve estimated input tokens, an output allowance, and concurrency against a provider and tenant budget. At this scale the arithmetic has to be explicit. If the mix averaged 2,000 input and 500 output tokens, 50,000 requests a second would ask for 100 million input and 25 million output tokens a second before retries. Those are illustrative numbers, not a measured workload. A three model plan is impossible unless its actual quota and capacity can carry the observed mix. Usage must be reconciled on completion, including partial streams and failures. Quotas may be measured in more than one unit, so request count alone is not an admission guarantee. Keep route assignment on each trace so a changed traffic mix does not get mistaken for a model regression.

The highest quality provider starts returning 429s. First determine whether it is a temporary rate limit, our own misestimated allocation, a particular organization or region quota, or a capacity problem on one route. Honor any provider retry signal within the request deadline, reduce new admissions to that provider, and use a shared retry budget so 50,000 clients do not all hammer it independently. OpenAI's rate limit guidance describes temporary 429 responses and Retry-After for its API. It is a concrete contract to inspect, not a reason to assume every provider's limit works the same way.

Now make the fallback decision per slice. If model two passed the same tool schema, grounded answer, and error mode tests for a low risk docs query, route new eligible requests there within its measured capacity. If the expensive model is required for a legal decision or a customer's pinned contract, queue only if the deadline permits, ask the user to retry, or fail clearly. Do not claim a successful response because a cheaper model produced plausible words. The fallback pool needs reserved or at least verified spare capacity. Dumping the entire failed route into it can create a second outage.

What if the 429 begins after the first tokens of a long stream? At that point swapping providers is not an ordinary retry. The user has already seen a partial answer and the replacement may disagree or repeat tool calls. End the stream with a clearly marked incomplete result or resume only through an explicit application contract that can preserve context, citations, and effect identity. Never replay a write capable agent call to another model as a generic networking fallback. The model proposes actions, but the runtime owns operation identity and authorization.

At 50,000 requests per second, control updates also matter. Route health observations need to be fast enough to prevent a stampede but stable enough not to oscillate between providers. Use bounded ramp changes, per provider and model quotas, and a canary for policy changes. Pinning, region restrictions, and high consequence exceptions must remain enforceable during control plane failure. A stale router may safely use a last known restrictive policy for a bounded time, but it should not invent a newly permitted provider.

I would evaluate the router on completed acceptable tasks within deadline per cost, split by tenant, slice, and chosen route. Test the 429 scenario with realistic prompt and output lengths, a pinned customer, a half finished stream, and a near full fallback pool. If the policy only works when every provider is healthy, it is a pricing algorithm, not production routing.