There are two separate questions hidden in “it loads.” Can the bytes be placed on the new ranks in a way the model and optimizer understand? And will training after that point mean the same thing as training before it? A loader can solve the first and leave the second entirely to us. PyTorch Distributed Checkpoint describes model and optimizer state dicts that can be resharded for a different number of trainers or parallelisms. That is a capability to use and test, not a promise that the training experiment is unchanged.

I would inspect the committed checkpoint manifest: exact model and optimizer configuration, global step, scheduler state, precision and loss scaler, RNG state, data sampler position, gradient accumulation phase, and the topology used to produce it. Create the 32-rank model and optimizer according to the supported loading protocol, then require a complete state load with no silently ignored keys. Confirm tied weights, partitioned optimizer moments and step counters match the expected global parameters. Run a small deterministic restore test on a checkpoint made for that exercise. A job that loads only weights and initializes fresh Adam moments may train, but it is not a continuation of the same optimizer trajectory.

Next calculate the training recipe. Suppose each rank previously processed 8 examples per microbatch with accumulation 4. The nominal global batch was 64 × 8 × 4, or 2,048 examples per optimizer step. On 32 ranks with the same settings it becomes 1,024. Doubling accumulation to 8 restores the nominal count, but the data ordering, random operations, batch statistics where used, and communication details may still differ. Preserve the learning-rate schedule in terms of optimizer steps or tokens as intended, not whatever the restarted job happens to count. If the accumulation checkpoint was taken between optimizer steps, we need either the in-flight gradient state or a deliberate rollback to a completed step.

Data coverage is another trap. A distributed sampler that divides records by rank can reshuffle the remaining epoch when world size changes. Record a global sample cursor or a reproducible shuffle plan. Otherwise a restart can repeat or skip training examples while still showing step 80,001 on the dashboard. In token-based training, track consumed tokens and packing policy, not only example IDs. Decide whether exact replay is required for this run. If it is, test that exact property instead of asserting it from a framework name.

Changing only data parallel degree is usually easier than changing tensor or pipeline parallel degree. The latter may alter partition shapes, execution order and kernel choices. The PyTorch FSDP checkpoint recipe provides a concrete loading pattern, but the trainer's parallelism implementation and state completeness decide what can actually move. Test the proposed 64-to-32 transition on a small representative checkpoint before risking the expensive run.

If all state and intended batch semantics are preserved, I would record a topology transition inside the run and continue with a verification window on loss, gradient norm, throughput and sample coverage. Bitwise identical updates across topologies are a much stronger promise and often unrealistic. If optimizer state or data continuity cannot be maintained, give the resumed job a new experiment identity with a documented warm start. Rank 7 dies while the training checkpoint is being written. Which checkpoint can you resume? asks whether step 80,000 is a complete checkpoint. Here it is complete. The problem is what that state means after the training topology changes.