I would ask how they got a score for a model that never answered that request. A log records the outcome of the old router's action. It does not tell us whether the cheap model would have solved the hard requests that the old router sent to the expensive model. Replaying the new router's choices over those logs can estimate its routing frequency and perhaps a list-price cost, but not its quality. The scored subset adds another selection mechanism: which answers did humans bother to score?

Write the estimand down. For a fixed eligible traffic population and cost accounting rule, compare expected useful, policy-compliant outcomes and spend under each routing policy. Record the request context available before the decision, the eligible models, chosen action, action probability if randomized, subsequent outcome, score inclusion probability, and censoring. A feature computed after the old answer is a leak, even if it predicts failure beautifully. List prices alone may omit retries, tool calls, latency, and escalations that change under a new route.

If the old router was deterministic, some request regions have no observations for the alternative action. No weighting formula can recover an unobserved outcome there. If an eligible exploration cohort in the logs did randomize with known nonzero action probabilities, inverse propensity or doubly robust estimates can help within that cohort, with uncertainty and sensitivity to model error. Dudík and colleagues' policy evaluation work gives the formal setting. I would show effective sample size and propensity distribution, not just a point estimate, because rare weighted observations can dominate. Human scores must be sampled or reweighted with their own selection process in mind.

The practical next step is to run both candidate models in shadow on a stratified sample, holding the request and allowed context constant. That gives paired outputs for expert review, but shadow cost and latency are not the same as a real route under production load. Also, a shadowed model must not execute external side effects. After policy and quality checks, randomize eligible live requests between the old and new router in a small cohort with stable assignment where conversations have state. Log the propensities and guardrails. Increase exposure only after measuring realized quality and total cost, including errors and fallbacks.

The interviewer may say every request from one enterprise customer was pinned to the expensive model by contract. Then there is zero support for estimating a cheap route on that customer's traffic, and the contract may make that route ineligible anyway. Exclude it from the proposed policy and from the savings denominator. A 30% claim built by routing contract-bound traffic cheaply is neither a quality estimate nor a feasible cost estimate.

Suppose they offer to have a judge score the cheap model on every historical request. That is useful evidence on paired response quality, subject to current source versions and the judge's calibration. It still does not recreate the consequences of using the cheap model live: users may ask different follow-ups, retries may change, and actions may affect later state. I would report that offline result for what it measures, then use a controlled online trial for the policy decision.