Interview scenario. A nightly pipeline writes 500 new Parquet files for the sales table. Two hundred files are uploaded successfully, then the coordinator crashes. A different job tries to publish an update to the same table at nearly the same time. The dashboard must see either the complete old version or one complete committed new version, never a random mixture of files.
Design the publication protocol and explain which component makes the new version visible. What becomes of uploaded files after a failed commit? What if the commit succeeds but the client loses the response?
We will use Apache Iceberg as a concrete implementation model, but the interview objective is broader: atomic visibility for a collection of objects when the object store does not offer a transaction spanning them.
Think first
A writer creates 300 Parquet objects and loses the catalog response after attempting the snapshot commit. The cleanup service sees those objects outside the current directory listing. Walk through the recovery decision without deleting possibly committed data.
Take a few minutes to form an approach, then compare it with the worked explanation.
Reveal the worked explanation
A directory full of files is not a table transaction
An object store can durably save individual objects. That does not make 500 separate uploads an atomic update. A reader listing a directory midway through the job could find only 200 new files alongside old files. It would have no reliable instruction about which objects jointly form a valid table version.
The naive design is to write the files directly under the live table path and tell readers to scan whatever is present. It works until a task fails, an old retry uploads another copy, or two writers run concurrently. There is no single commit event that tells the reader which collection of files is authoritative.
Instead, treat the data files as immutable material that is not visible merely because it exists. Write the files first. Construct metadata describing the complete new table state. Publish that metadata with one atomic table-level decision.
In Iceberg, a snapshot describes a set of data and delete files through manifests and manifest lists. A table metadata file points to its current snapshot. A catalog or supported table implementation supplies the atomic change of the table metadata pointer. Readers pin a committed snapshot and plan from that metadata rather than discovering arbitrary uploaded objects. Iceberg specification
Immutable data-file uploads lead to new metadata and an atomic catalog pointer that lets readers pin the old or new snapshot.
Work through the crash
Start with committed snapshot S40, containing the valid sales data. The writer builds data files for S41 in object storage. Those files have unique names and do not overwrite the files used by S40.
At the moment the writer crashes after file 200, the catalog still identifies S40 as current. A reader starting now sees S40, not 200 of the intended 500 new files. If the writer finishes all 500 but crashes before committing the new metadata pointer, the same rule applies. Uploaded files alone do not make a table version valid.
A successful commit changes the table's metadata reference to a new metadata file naming the committed snapshot. Readers that have already planned on S40 can continue reading it, subject to normal retention and reader-lifetime protections. New readers observe the new committed metadata after refreshing the catalog.
That division is the central mechanism. The object store holds immutable bytes. The table metadata and atomic pointer establish which bytes count as a visible table version.
Two writers can both be correct locally
Suppose a compaction job started from S40 and rewrote five old files. At the same time, the nightly ingestion job built S41 and committed first.
The compactor cannot simply replace the catalog pointer with a metadata tree based on the old snapshot. Doing so could discard the ingestion update. The commit must compare the expected base with the current table metadata and apply the operation's validation rules. A conflicting rewrite may need to refresh the table, verify the affected files still exist, and retry or abort. The Iceberg spec describes optimistic concurrency and notes that isolation depends on what each write validates. Iceberg specification
"The catalog uses compare and swap" is not the whole answer. The new metadata must represent the intended full table state, and the conflict rules must protect the logical operation. Blindly retrying a stale full overwrite can silently lose another writer's changes.
Also distinguish table atomicity from transactions spanning multiple tables. A single-table snapshot commit does not automatically guarantee one atomic cutover across both sales and inventory tables. If a report joins several tables and needs a consistent point in time across them, it requires an additional snapshot-selection or transaction contract.
A lost commit response is a serious edge case
The writer sends the catalog update, and the response disappears. It does not know whether its new snapshot became current. Publishing again from scratch could duplicate an append. Deleting the uploaded files could corrupt a committed table.
The recovery path should inspect commit status and the table's current metadata and history using stable operation identities. It should resolve whether the intended snapshot was committed before choosing to retry or clean up. Iceberg exposes commit-status checking options precisely because the outcome can be uncertain when the communication fails. Iceberg configuration
Cleanup follows separate rules. Files belonging to an in-progress writer must not be removed just because the current pointer does not yet reference them. Old data files remain as long as retained snapshots need them, and orphan cleanup requires a cautious retention interval. Iceberg maintenance
In an interview, I would close by naming the boundary explicitly: the 500 uploads are preparation, and the atomic table metadata update is the publication decision. A distributed batch engine may retry tasks many times, but readers should only adopt a coherent committed snapshot.
That protects the structure of the table. It does not prove the rows are meaningful. Our final question concerns a job that publishes one perfectly atomic, completely wrong answer.
Go deeper
Optional depth: visibility is not automatic deletion
Immutable snapshots allow readers to use an older version while writers publish a newer one. That benefit depends on retention. Expiring snapshots too aggressively can break long-running readers or time-travel queries. A safe implementation coordinates reader expectations with table retention and storage cleanup. It must also account for delete files and concurrent rewrite validation, not only append-only data files.
Check your reasoning
Distinguish file upload from the atomic table commit.
Describe immutable data, manifests, snapshots and catalog pointer publication.
Treat concurrent writer conflicts and unknown commit outcomes explicitly.
Keep uncommitted and old snapshot files until safe to remove.
Common mistake: Completing all object PUT requests automatically publishes one atomic table version.
Read the answer explanation
The committed S40 snapshot — Object existence alone does not publish S41.
Apply it elsewhere
A search index build uploads segments before changing an index-generation pointer. Which behavior should be shared?
Compare your approach
Readers need a stable generation and maintenance needs retention-safe reclamation of old segments.
One more constraint
If the update is atomic for one Iceberg table, can a reader assume a simultaneous update to another Iceberg table is part of the same transaction?
Quick revision
Saving many objects is not an atomic table update. The writer uploads immutable files, builds a complete snapshot in metadata, and atomically publishes the table's metadata pointer through a supported catalog. Readers use one committed snapshot. Concurrent writers must validate conflicts, an unknown commit result must be resolved before retry or cleanup, and old files remain until retention safely permits deletion.
