The missing receipt proves very little
An old event e901 arrives on day 30. The handler checks its processing-receipt table, which no longer contains e901. It treats the event as new and performs the effect again.
Two histories are now indistinguishable to that lookup:
A. e901 never completed
B. e901 completed, but its receipt expired
Receipt lookup in either case: not found
An absent receipt does not prove an effect never happened. Starting a new Kafka consumer group creates a new reading position. It does not create a new legitimate business operation. And changing fulfillment-live to fulfillment-replay may bypass the very receipt scope that made repeated delivery safe.
This is a different problem from the 48-hour retry protection we designed earlier. Here, an operator is deliberately introducing old records into a path that can still change the world.
Rebuilding data is not repeating an action
For the dashboard, I would not run the warehouse or email handler at all. I would build a restricted projection worker that reads historical facts and writes only to a new dashboard generation. It has no credentials to reserve inventory, no permission to send notifications, and no ability to publish fulfillment commands.
That separation matters more than a boolean called replayMode buried in application code. If someone accidentally invokes the wrong method during a backfill, the execution environment still cannot ship old orders.
Say the corrected view is daily-order-totals-v2. Every historical order contributes a deterministic record keyed by the view generation and original event ID. Duplicate archive delivery does not add its amount twice. The input must contain the facts the calculation requires, such as accepted amount, currency, and event date. Looking up today's catalog price would answer a different historical question.
Nothing in this requires the whole business system to be event sourced. It requires sufficiently complete history for the particular derived view. If required facts were never recorded, replay cannot invent them.
Catch up without disturbing live readers
First capture an archive boundary for each source partition and a manifest describing source ranges, schema versions, transformation version, output generation, and checkpoint positions. Historical records are applied to the new projection while the old dashboard continues serving requests.
When the archive catch-up reaches its boundary, continue from the appropriate positions in the live stream. If archive and live data overlap, use original event IDs to suppress repeated contributions. If there is a gap, stop and report it. Kafka partitions have separate offsets, so “replay to offset 900” is not a sufficient cross-partition boundary.
Do not give the backfill all the capacity it asks for. It may share database disks, network bandwidth, or broker capacity with live traffic. Throttle against those actual bottlenecks. A separate consumer group protects reading progress but does not provide resource isolation.
Before switching the dashboard, compare totals by day and currency, sample known orders, verify covered source positions, inspect rejected records, and check whether the new view meets the freshness objective. An overall total can appear correct while missing one partition and double-counting another.
Only then change the dashboard's generation pointer. Keep the old generation for a defined recovery window. Rollback is possible only while the old view remains compatible and sufficiently up to date.
Four different kinds of memory
The checkout operation record, the broker's send-deduplication window, the consumer's generic receipt, and the warehouse's durable business fulfillment record are not interchangeable. They may have different lifetimes because they defend against different problems.
A short receipt window can be appropriate for ordinary redelivery. It cannot protect a privileged process that may replay any event from years ago. Increasing retention may help, but it still does not make it safe to resend irreversible emails or reapply changed business rules.
What if the request is different: “We found 200 accepted orders that genuinely never reached fulfillment. Finish them”? That is reconciliation, not analytics replay. Consult the authoritative fulfillment outcome for each business identity, preserve the original operation IDs, and follow its recovery protocol. Do not send every historical event through the live handler and hope the receipt table sorts things out.
The interesting question in an interview is therefore not just how to read old offsets. It is which historical facts may be recomputed and which real-world actions must never be repeated. Enforce that boundary with the worker's permissions and output contracts, then make completeness and cutover measurable.
One more failure can erase the evidence without waiting 48 hours. A new region can be promoted from a replica that never received the accepted order. What happens to all these guarantees then?