The evaluated population is “transactions whose outcome became known within seven days,” not all transactions the model scored. If label arrival depends on the outcome, amount, customer segment or the model's own interventions, the sample is selected. An unlabeled payment is not a negative payment. Calling it negative too early makes later fraud look like a sudden model regression when it is partly a measurement artifact. Even on a mature set, accuracy can hide rare but costly fraud behind a large honest class. Delayed-feedback research in conversion prediction describes how positives observed later can be mislabeled as negatives during training or evaluation, for example in this paper. A chargeback system has its own outcome process, which must be studied rather than imported from advertising.

I would store each prediction with a stable transaction ID, model version, score, decision, timestamp, eligibility and any action taken. Join the eventual label by ID and retain both event time and label-observed time. Databricks inference tables describe request identifiers that can support this kind of later ground-truth join. The important point is the cohort definition, not the particular logging product.

Then evaluate mature cohorts whose label window has mostly closed, and report what fraction remains unresolved. For recent cohorts, show provisional metrics with a known observation horizon, not a final accuracy number. If we use survival methods or inverse-probability weighting to estimate current performance, the assumptions about label delay and missingness must be checked by outcome and segment. Unknown fraud may never be discovered. A statistical correction cannot create ground truth that the business never observes.

There is a second trap. The model's decision changes which outcomes we can see. A declined payment cannot later produce a chargeback, so comparing chargeback rates among approved transactions is not an unbiased comparison of both models. Randomized holdouts or carefully designed review samples may be needed to estimate false declines and downstream harm, within product and safety constraints. Report coverage, approval rate, fraud loss, false-decline review and customer impact by segment, not just accuracy on the easy-to-label set.

Suppose product needs a decision this week. I would use mature prelaunch cohorts and leading indicators for the new model, plus a guarded online experiment with explicit loss limits. I would not pretend a seven-day label is a completed six-week outcome. Set the date when the result can be revisited, and define what would stop the rollout before then. The answer changes if most labels arrive by day two and the lag is stable, but we need that empirical delay distribution and its confidence by cohort.

The training feature knew about a payment that had not arrived yet. How? deals with information that arrived after the prediction and leaked into training features. Here the input can be perfectly point-in-time correct while our evaluation is wrong because the outcome has not arrived yet. Those are different clocks and different fixes.