Data and Knowledge Systems · Principal
A bulk snapshot races the change stream
The question
Interview question
Assume a connector is exporting a hypothetical 200 million source objects into a new search index before consuming changes. During export, one object is edited and another is deleted. A late snapshot record overwrites a newer change. Design a gap-free bootstrap. The change log may expire before the export ends.
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
“Snapshot first, then turn on CDC” leaves a gap unless the snapshot and log have a consistent handoff point. Starting CDC early avoids that gap, but snapshot rows can arrive after newer updates and deletions. Arrival order at the index is not source order. If a delete is followed by a stale snapshot write, a document comes back from the dead.
Use a source mechanism that relates a consistent snapshot to a durable change position. Start or retain the stream at a position that covers writes concurrent with the snapshot, capture the snapshot under a source-consistent view, and apply both into a staging generation. For each source object, store a version or commit position and make updates monotonic. A snapshot read cannot replace a newer streamed update or tombstone. If the source cannot attach comparable versions to snapshot and change records, the connector has to coordinate their collision window explicitly. Debezium's PostgreSQL incremental snapshot documentation describes chunk windows, watermarks, and deduplication when a streamed update or delete races a later snapshot READ. That is one implemented pattern, not a promise that every connector has those guarantees.
The source commit, extraction, and index activation are different clocks. Record the snapshot position and coverage manifest, the earliest retained stream position, per-partition catch-up watermarks, and parse or embedding job status. Do not promote when only the event consumer says it reached the end of the log. Check that every snapshot partition was completed, the stream has no gap, and index writes and deletes through the chosen watermark are durable and queryable. Keep authorization at disclosure time as in DKS-001. A complete content index built from an old ACL does not give current permission.
Now let the log expire midway. If the required starting position is gone, replaying from the earliest available event cannot prove that no edit or delete was missed. Stop promotion. Take a fresh source-consistent snapshot with a new stream handoff, or reconcile the staged generation against an authoritative full inventory that includes existing identities, versions and deletions. A spot-check of a few documents is not a completeness proof. If the source offers neither a consistent snapshot, a durable feed, nor an authoritative inventory, state that the requirement cannot be met from its interface. The right failure mode is an incomplete or unavailable new generation, not a plausible index with unknown missing changes.
Continue practicing
Related questions
Read beyond the question
Explore more data and knowledge systems
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →