Imagine a mixed text and code training job. Code records take longer to fetch and tokenize. To keep GPUs busy, the team lets whichever worker finishes first deliver the next batch. That can be a good throughput choice, but now completion time helps decide which samples arrive early. If training stops after a fixed number of steps, restarts often, or draws a fresh prefetch queue every epoch boundary, slow records can be underrepresented among consumed tokens even though the source manifest has the right mixture. PyTorch's DataLoader documentation warns that in_order=False can harm reproducibility and skew the data fed to the trainer when the input is imbalanced. The exact bias in this scenario needs measurement.

I would count consumed sample IDs and tokens by source, length and processing-time bucket, not just queued or scheduled examples. Record enqueue, worker finish, batch consumption and drop or cancellation events for a small run. Compare the expected manifest distribution with the first N optimizer steps under ordered and unordered loading. If all records are eventually consumed exactly once before a full epoch finishes, reordering alone changes sequence order but does not necessarily change the epoch's final counts. It can still affect training dynamics. The stronger mixture bias needs a stop, truncation, retry or queue-reset boundary that actually discards or postpones slow samples.

To keep the speedup, assign a deterministic sample schedule and bound out-of-order completion, or account for completed-but-not-consumed records across checkpoints. Persist sampler position and pending work consistently. Another option is to balance preprocessing costs across workers or pretokenize the expensive slice. Verify both achieved throughput and token mix on a fixed training budget, then compare evaluation on the slice that was previously delayed.

"But the random seed is fixed" is not enough. The seed can fix which samples were scheduled while worker timing decides which ones the optimizer saw before the stop. Every training rank has a different shard. Why are its random augmentations identical? is about identical augmentation randomness across ranks. This one is about the delivered training distribution when loader completion order becomes part of the selection process.