Because the batch is part of the loss, not merely the optimizer schedule. For a query, an in-batch contrastive loss compares its positive document with the other documents in the same similarity matrix. With 256 pairs in the forward pass, there are up to 255 other document candidates. With 16, there are 15. Sixteen separately computed losses do not equal one loss on a 256 by 256 similarity matrix, even if their gradients are accumulated and the optimizer steps once. The denominator, the hard competitors, and therefore the gradient direction have changed.

I would make this visible with one frozen 256-pair batch. Run the original loss once, then the 16-way microbatch recipe on exactly those pairs. Compare the similarity matrices and which negative each query finds hardest. If the first recipe also gathers embeddings across data-parallel ranks, verify whether the new recipe still does so at the same boundary. A distributed gather can expand the candidates within one forward pass. It does not automatically join examples from future accumulation steps.

To preserve the large candidate set without encoding all 256 pairs with activations live at once, a gradient-cache approach can encode mini-batches, construct the larger embedding-level loss, then replay mini-batches for parameter gradients. Sentence Transformers documents its cached multiple-negatives loss for this memory tradeoff. Or choose a smaller candidate set deliberately and compensate with mined negatives, after measuring whether that actually recovers the desired retrieval behavior. Watch for false negatives too. Another query's valid document is not necessarily a safe negative, and simply increasing the matrix can amplify that problem.

If someone argues that both runs process exactly 256 pairs per optimizer step, I would agree with the arithmetic and show the two denominators. The optimizer sees accumulated gradients from 16 different classification problems, not the gradient of one 256-way problem. We doubled gradient accumulation after losing GPUs. Why did the training recipe change? studies accumulation changing a language-model training recipe through token weighting and schedules. Here the identity of the negative examples inside a contrastive objective changes before the optimizer sees any gradient.