The training feature knew about a payment that had not arrived yet. How?

We need two clocks. The payment's event time says when it happened. Its availability time says when the scoring system could have known it. At 10:00 yesterday, the live model might have seen four payments. Today's reconstructed table shows five payments with event times before 10:00. A join that only checks event time has made yesterday's model clairvoyant.

This is a subtle distinction because a normal point-in-time join does prevent one kind of leakage, where it takes a feature event from after the decision. It does not necessarily prevent a backfill written later with an earlier event timestamp from entering the historical result. Feast's point-in-time join documentation explicitly distinguishes this case. Its created timestamp can break ties between records with the same event timestamp without generally limiting the join to what had arrived by the decision time. Do not infer availability filtering from a column name.

I would reconstruct the actual serving view for a sampled decision. Store the decision ID and timestamp, feature definition version, materialization or stream watermark, value returned, missing-value fallback and, where permitted, the feature record's observed-at revision. Compare that with the offline row for the same decision. If the count differs, work backward through late arrivals, corrections, TTL, timezone boundaries and duplicate event handling. A model-quality dashboard by itself will not tell us which join changed.

For a reproducible training set, define the prediction cutoff precisely. Join records whose event time is within the intended window and whose availability is no later than the cutoff, or replay a versioned online-state log as it existed then. If ingestion time is just when a warehouse copied an already available online event, it is the wrong availability clock. We need the first time the serving path could actually use that value. Sometimes that clock was never recorded. Then we cannot claim an exact historical replay. We can use prospective shadow logging to build a clean set and label older analysis as an approximation.

The online and offline feature definitions should share the same semantic contract: entity key, time window, inclusion rules, handling of reversals, late events, missing values and freshness limit. Sharing source code helps, but does not fix an online stream that drops late events while offline recomputation includes them. Feast feature retrieval describes the different historical and online retrieval paths. Test their outputs on the same timestamped cases, including a late event and an event corrected after the decision.

What if a late payment is genuinely useful for tomorrow's decisions? Of course we should include it when it becomes visible. The rule is about what yesterday's model knew yesterday, not whether the event must be discarded forever. Recompute current aggregates with the correction if policy permits, keep the old decision's observed feature immutable, and make any later evaluation distinguish original decisions from hindsight reconstructions.

Before restoring traffic, I would compare the new training set with logged serving values by cohort, inspect feature null rates and delayed-event rates, retrain if the leakage materially changed the target relationship, then run a shadow period and a guarded rollout. The decisive test is simple: can I take one past prediction and show the exact information that was available at that instant? Three data sources are current and one is a day behind. Is the AI report current? asks when a report with several sources is fresh. Here the source can be historically complete yet still wrong for training a model that made decisions in real time.