A connector event is replayed after the file changes. Which version gets indexed?

An event ID tells us which notification was replayed. It does not tell us which bytes a later fetch returned. Suppose the connector emitted “file F changed at revision 11.” Before a worker fetches it, the file reaches revision 12. If the source only serves its current state, the worker gets revision 12. Calling those bytes revision 11 would make every later deduplication and citation check suspect.

There are two workable source contracts. If the source lets us fetch immutable revision 11, process that snapshot and give the result a source identity such as (tenant, source, file ID, revision 11). If it only lets us fetch the latest, treat the event as a dirty signal. Fetch the current bytes and metadata together as consistently as the source permits, record the observed revision or ETag, and process that. A content hash proves which bytes we stored, but it does not tell us whether one revision is newer than another.

I would make the processing output immutable: source snapshot ID or observed revision, content hash, parser version, chunker version, embedding version, and the resulting chunk manifest. Duplicate workers may do some work twice after a crash. They must produce the same logical target for that version and cannot both create distinct active copies. The document registry holds the desired source state and a pointer to the last complete searchable version. Preparation can be retried. Moving the active pointer needs a conditional check against the current desired version and deletion state.

The out-of-order follow-up is where this check earns its place. Worker B finishes revision 12. Worker A, which started on revision 11, finishes later. A's chunks can remain as historical artifacts if the product needs them, but A cannot replace the active pointer for 12. A late revision 10 delete or upsert cannot resurrect an older document either. DynamoDB conditional writes are one implementation mechanism for that compare-and-set. The ordering rule still has to come from the source or our reconciled source state.

What if revisions are opaque ETags with no ordering? I cannot decide that abc9 is newer than aef2 from the string. Use a monotonic change sequence if the source provides one, or fetch the authoritative current state during reconciliation and compare the fetched identity to the work being activated. If the source can change between fetch and activation, the contract needs a bounded freshness window or another current-state check. An ingestion timestamp is not a substitute for source order. The late worker may have fetched an older snapshot after a newer event was observed.

There is also a delete race. A stale upsert can arrive after a delete notification. Record the delete in the desired-state ledger and block old versions at serving time before background index cleanup. If the source can restore the same file ID later, distinguish the restored revision from the old one. Otherwise a simplistic permanent tombstone will hide a legitimate restoration, while a simplistic “latest worker wins” will expose deleted text.

Take away immutable versions and the reliable ordered feed. Then I would not claim an exact event-by-event history. I can keep a latest-state index by periodically reconciling the source and recording the observed bytes and checks, with an explicit detection lag. If the source cannot provide an authoritative current view either, there is no basis for a strict freshness promise. That limitation belongs in the product contract, not in a comment next to a retry loop.

I would test the worker after fetch, after writing half the chunks, and just before activation. Replay the event at each point while the source advances and while a delete occurs. The visible answer should cite either the last complete permitted version or the newly activated one, never a mixture assembled from whatever work happened to finish first.