Model and Inference Engineering · Principal
The effective batch stayed at 256. Why did the retriever lose most of its negatives?
The question
Interview question
An embedding model was trained with 256 query-document pairs in each forward pass. To save memory, the team changes to microbatches of 16 and accumulates gradients for 16 passes. The optimizer still updates once per 256 pairs. Retrieval quality falls. Why is the effective-batch calculation misleading here?
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
Because the batch is part of the loss, not merely the optimizer schedule. For a query, an in-batch contrastive loss compares its positive document with the other documents in the same similarity matrix. With 256 pairs in the forward pass, there are up to 255 other document candidates. With 16, there are 15. Sixteen separately computed losses do not equal one loss on a 256 by 256 similarity matrix, even if their gradients are accumulated and the optimizer steps once. The denominator, the hard competitors, and therefore the gradient direction have changed.
I would make this visible with one frozen 256-pair batch. Run the original loss once, then the 16-way microbatch recipe on exactly those pairs. Compare the similarity matrices and which negative each query finds hardest. If the first recipe also gathers embeddings across data-parallel ranks, verify whether the new recipe still does so at the same boundary. A distributed gather can expand the candidates within one forward pass. It does not automatically join examples from future accumulation steps.
To preserve the large candidate set without encoding all 256 pairs with activations live at once, a gradient-cache approach can encode mini-batches, construct the larger embedding-level loss, then replay mini-batches for parameter gradients. Sentence Transformers documents its cached multiple-negatives loss for this memory tradeoff. Or choose a smaller candidate set deliberately and compensate with mined negatives, after measuring whether that actually recovers the desired retrieval behavior. Watch for false negatives too. Another query's valid document is not necessarily a safe negative, and simply increasing the matrix can amplify that problem.
If someone argues that both runs process exactly 256 pairs per optimizer step, I would agree with the arithmetic and show the two denominators. The optimizer sees accumulated gradients from 16 different classification problems, not the gradient of one 256-way problem. We doubled gradient accumulation after losing GPUs. Why did the training recipe change? studies accumulation changing a language-model training recipe through token weighting and schedules. Here the identity of the negative examples inside a contrastive objective changes before the optimizer sees any gradient.
Continue reading
Related questions
Read beyond the question
Explore more model and inference engineering
Follow another question in this area, or search the complete Question Library.
Browse this area →Browse Question Library →