Rank 7 dies while the training checkpoint is being written. Which checkpoint can you resume?

I would restart from 78,000 unless the checkpoint system can prove 80,000 is complete and coherent. The presence of most shard files is not that proof. A single missing optimizer partition may prevent loading, but even a loadable collection of files can be wrong if ranks captured different steps or the scheduler and data cursor came from another moment. A checkpoint is a consistent training state, not a directory with a plausible number of files.

The state includes model parameters, optimizer moments and step, learning-rate scheduler, loss scaler if mixed precision uses one, random number generator states, data sampler or loader position, and any training-specific state such as curriculum phase or accumulated gradients. Which of these can be reconstructed depends on the training algorithm and tolerance for replay. If we resume from 78,000, replaying a few samples may be acceptable in one run but not in a run that promises exact data coverage or deterministic debugging. We should say which guarantee we are making.

I would give every save a unique generation ID and a fixed step boundary. Each rank writes its shard under an uncommitted prefix, with size and digest. A coordinator collects completion from all required participants, checks step and topology metadata, then publishes a small immutable manifest or commit marker. Readers enumerate committed manifests, not raw prefixes. If rank 7 dies before commit, the partial generation is garbage for resumption and can be cleaned later. If the coordinator dies after writing the manifest but before reporting success, the manifest is the durable authority. A second coordinator should verify it and treat the operation as complete rather than publishing a conflicting version.

This is an application-level protocol. PyTorch Distributed Checkpoint supports sharded checkpointing and resharding, but using an API does not automatically make arbitrary trainer state and object-store publication coherent. The PyTorch tutorial demonstrates how model and optimizer state are collected for a distributed save. We still need to include the data and schedule state for our training loop and verify the actual storage semantics.

If the interviewer says all 64 weight shards finished but one optimizer shard did not, I would not silently resume only the weights and call it the same run. That becomes a new training experiment with reset optimizer state, potentially a different loss trajectory. It may be useful as an explicit recovery experiment, with new run identity and validation. It is not a faithful continuation. If asynchronous saves read tensors while training proceeds, we also need a snapshot boundary or staging copy so shard data cannot represent different update steps.

Test the crash before any upload, during one shard, after all shards but before commit, and after commit but before acknowledgement. Restore the checkpoint on a separate job, run several steps, and compare a small deterministic fixture where feasible. Monitor time since last committed and restorable checkpoint, not time since the last save started. A million-row model batch fails at 70 percent. What exactly can you publish? covers publishing a partial model batch to readers. Here the publication object is the training state from which all workers will continue.