I would first say what “helps” means in the actual product. The first reply is an early signal, not the outcome. I want an issue resolved correctly, with the right user authorization, without an avoidable handoff, duplicate action, or later reversal. Time to resolution matters, but faster wrong resolution is worse. Cost, latency, user effort, safety incidents, and human escalation should be reported beside that outcome, not hidden inside one score.

Randomizing individual turns would break the experiment. A conversation could switch agents halfway through, and the later answer inherits earlier memory and tool writes. Assign a stable unit before the first eligible exposure and keep its assignment across the issue and follow-ups. If memory and quotas are shared across a tenant, tenant level randomization is the clean default. It reduces sample size and can make large tenants dominate, so stratify by tenant size and pre-period activity, then analyze with the tenant as the independent unit. If the product can isolate memory and quotas by user, and users do not work on the same issues, a stable user assignment may be defensible. That isolation claim needs checking, not assuming.

Shared quota creates an additional problem. If treatment uses more API calls, a control issue in the same tenant could be rate limited. The control outcome is then affected by treatment. Tenant assignment removes that particular within-tenant spillover. Cross-tenant shared capacity may still cause it. Monitor saturation and consider reserving capacity per arm or staggering exposure. The point is to measure the whole policy, including resource use, under an experiment where one arm cannot quietly change the other's environment. Research on interference in collaboration experiments is relevant here: independent per-person assignment is not enough when outcomes depend on neighbors' treatment.

Define the denominator before launch: all eligible issues assigned at entry, including ones where the agent abstains, escalates, or never reaches a final answer. Track an issue ID through reopening and follow-up. Resolve outcomes on a fixed horizon such as 14 or 30 days, chosen from the product's resolution distribution, and report the still-open fraction at that horizon. A week-one dashboard that drops unresolved issues preferentially can reverse the comparison. Microsoft's work on long-term online experiments discusses these selection and survivorship traps.

I would start with a small ramp, a predeclared primary outcome, and guardrails on unauthorized writes, wrong recipients, reversals, severe factual errors, tail latency, and cost. Log assignment, agent version, memory scope, tool and policy versions, and actual exposure. Analyze by original assignment for the primary estimate. An “only completed conversations” analysis can be diagnostic, but it is not a substitute if completion itself changes under treatment. For rare severe events, a short A/B test cannot prove safety. Keep incident review and hard policy checks in the release gate.

Suppose the interviewer says the tenant sample is too small and asks for turn level randomization. I would trade duration or scope before accepting contamination. We might target new tenants, isolate state for a subset, or use matched tenant blocks and a longer run. If assignment changes the shared world, a bigger count of turns gives a more precise answer to the wrong question.

One more twist: an issue can be reopened after the measurement window. I would keep the original windowed result for the predeclared analysis, and separately report longer-horizon reopen and reversal curves as they mature. Do not silently relabel past results as late events arrive. The release decision can wait for a minimum amount of mature outcome data if reversals are a central product risk.