Ten times more records can be less diversity if they are copies of the same templates. Sampling happens over examples or tokens, so repeated material gets more gradient weight. The model can get better at predicting common phrases while receiving a smaller fraction of updates from rare workflows. A lower average loss is real for the mixture it saw. It does not prove the mixture represents the customers we care about.

I would freeze the run and compare the actual token stream, not just the count of source files. Measure exact and near duplicates, long repeated spans, per-source token share after filtering, language and customer distribution, document age, and overlap with validation. Calculate the effective contribution after sampling and packing. A corpus that is ten times larger on disk may be much smaller after deduplication, or it may overwhelm a carefully chosen rare slice when we sample proportional to size. The ACL paper on deduplicating language-model training data demonstrates that near duplication and train-test overlap materially affect training and evaluation. It does not say that every duplicate in our corpus is bad. Some repetitions reflect actual user frequency and may be useful.

We need separate hypotheses. Did the new data dilute rare workflows? Did it contain stale support advice? Did it leak validation items? Did evaluation improve because the new corpus repeated test phrasing? Did the model genuinely forget an old capability? Build fixed, untouched slices for affected customers and generic tasks, with source-independent examples where possible. Compare the prior checkpoint, the new checkpoint, and a small controlled continuation with changed mixture weights. Keep decoding and prompts fixed. Inspect errors rather than reporting a single average.

If the interviewer says deduplicate everything globally, I would push back. A normalized error code appearing in many distinct cases is not the same as a copied article. Removing every similar sentence may erase exactly the domain signal we need. Use document-level and span-level audits, protect rare legitimate examples, and set a deliberate sampling policy. We can cap extreme repeats, sample by source or task, and preserve a minimum share for rare but important workflows. Record the recipe so the run can be reproduced.

There is also an attribution problem. Maybe the new model simply received more total training tokens or a different learning-rate schedule, and the corpus is not the only change. A controlled ablation holds compute and schedule as comparable as practical while changing the mixture. If we cannot rerun at full scale, use smaller pilots to reject obvious failure modes, then confirm a selected recipe on a production-sized run. Evaluate source-specific loss, task success, memorization probes where appropriate, and degradation for named cohorts.

I would ship nothing based only on aggregate loss. The decision is a mixture that meets quality floors for the important slices, not the largest pile of text. You tuned against the holdout for six months. Is it still a test set? asks whether a repeatedly tuned holdout is still a test. This question asks whether the training distribution gave repeated common examples enough weight to hide a real minority regression.