Data and Knowledge Systems · Staff
The CDC consumer is healthy again. Did it miss three days of changes?
The question
Interview question
A connector mirrors account records into an AI search index. It was down for three days. Its change stream retains one day. On restart the client resets to the oldest available event, catches up to current time and reports zero lag. Search still returns old account status for some customers. How do you detect the gap, repair it without dropping changes during repair, and decide when answers can be trusted?
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
Zero lag only says the consumer reached the current end of the log from wherever it restarted. It says nothing about the two days already trimmed. A stream's retained horizon is not a complete history. For example, DynamoDB Streams documentation describes finite retention, and its shard iterator API distinguishes starting after a sequence number from the oldest untrimmed record. The scenario does not depend on DynamoDB specifically. Any CDC source with bounded retention has this failure mode.
Store a durable per-shard or per-partition checkpoint and compare it with the source's earliest available position on restart. A checkpoint earlier than retention is an integrity failure, not a normal reset. Raise an alert, mark the affected index generation incomplete and stop claiming current coverage. Retain the evidence of the missing interval. If the client silently resets, add a guard around it. A heartbeat or current offset cannot prove that every earlier mutation was applied.
Repair needs a consistent boundary. One approach is to take a source snapshot with a cursor or version that the source guarantees corresponds to it, then replay changes after that cursor into a new index generation. The exact mechanism varies. A snapshot and a stream cursor taken at unrelated moments can miss writes between them or double-apply them. If no atomic snapshot-plus-cursor interface exists, use a documented overlap and reconciliation protocol, stable record versions and idempotent upserts and tombstones. Do not assume wall clock timestamps totally order changes across shards.
Build the replacement generation while the old one still serves under restricted guarantees. Apply the snapshot to an empty generation, then catch up from the captured change boundary, including subsequent deletes. If repairing in place, compare old IDs with the snapshot and remove rows absent from current source state. A snapshot of current rows does not itself list all historical deletes. Compare counts and sampled record digests by partition with the source, and inspect high-risk records changed during the gap. If deletion or permission changes might have been missed, a current authoritative serving check must deny stale index text, or the affected corpus must be withheld. The Delta table deleted a customer row. Why does vector search still return it? describes that serving denial for a known deleted row. This question asks how to recover when the consumer cannot even enumerate the missed events.
At cutover, fence writers for the old generation, verify the new generation's applied positions and coverage, switch reads atomically, then continue streaming without leaving a window. If repair takes longer than retention, retain the required stream interval or repeat the snapshot protocol. Increasing retention now does not restore already expired events. If the source no longer has historical versions, we may recover current state but not every intermediate transition. That distinction matters for audit, derived aggregates and triggers. An index of current account status can often be rebuilt from a current snapshot. A ledger of every status transition cannot be recovered from current rows alone.
Test a deletion during the gap, a write during snapshot, a crash before cutover, shard split, out-of-order retry and a source record missing from the new index. The recovery completion signal is source reconciliation and proven coverage at a named boundary, not a dashboard showing zero stream lag.
Continue reading
Related questions
Read beyond the question
Explore more data and knowledge systems
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →