I would ask what “win” means before explaining the difference. Is it a user completing a task, a thumbs-up, a human preference comparison, or an automated judge over a sampled answer? What is the denominator? A drop in feedback per exposed session can come from different people responding, even if the answers did not change. An offline score may be stable because its frozen cases never contained the new traffic.

If this is a randomized rollout, I want the assignment and exposure trail. Which user or session was assigned, which prompt version did the request actually use, and which requests produced an outcome event? A cache or a long-running agent can keep using an old prompt after a new assignment. A failure before feedback logging can selectively remove sessions from the measured denominator. First check the intended split against the observed split and the missing-event rate. Microsoft's sample ratio mismatch work is a useful reminder that a broken experiment can make either a good or a bad change look like the opposite.

Then inspect paired traces around the first bad step. Did the new prompt change retrieval queries, tool choices, answer length, abstention, or latency? Were the same model snapshot, index generation, policies, and traffic routing used in both arms? A prompt change can reduce task completion because it asks too many clarifying questions while still receiving a good offline judge score for careful language. Or the online metric can fall because a UI event stopped firing in one client version. Both need evidence.

The traffic mix follow-up matters. Suppose the week brought more version-specific enterprise queries. A frozen eval dominated by simple public docs questions stays flat, and the new prompt may fail badly on the enterprise slice. I would compare treatment and control within the same concurrent cohorts by pre-exposure attributes such as query family, tenant tier, client, language, and region. Then recompute an aggregate on a common reference mix. Do not compare this week's treatment against last week's control and call the prompt causal. Also avoid defining a slice using a behavior the treatment itself changed, such as “sessions where the model chose to answer.” That can bias the comparison.

If assignment was sound and the within-slice regression is real, I would add representative cases from that slice to a held-out offline set and inspect why it missed them. Perhaps the offline labels only score factuality, while online users need the agent to finish the workflow. Perhaps the eval replays old retrieved passages and never tests the new prompt's search behavior. I would keep separate scores for retrieval, decision, final support, tool outcome, and the user result. Flatness at one boundary does not veto a regression at another.

Suppose the online drop is statistically noisy. I would look at its uncertainty and consequence. A small apparent change on a low-risk path may justify holding the canary while we gather data. A few confirmed permission leaks or wrong financial actions justify stopping exposure without waiting for a neat confidence interval. The incident response can pause rollout on credible harm while the causal analysis remains open. Those are different decisions.

I would only call the prompt the cause after checking assignment, exposure, logging, traffic mix, and other changes during the window. Then rerun the corrected comparison and a targeted offline suite. If the prompt regressed one slice, either fix it and canary again or keep that slice on the old version. If instrumentation caused the apparent drop, repair the metric before celebrating a rollback. The goal is to know what users actually experienced, not to pick the dashboard we prefer.