I would stop the job at a completed step and find out exactly what changed. The model sees token IDs, not the text we think we fed it. If an old tokenizer maps a word to IDs 41 and 902, while the new one maps that word differently, the embedding rows and the optimizer moments attached to those rows retain their old history. Even with the same vocabulary size, IDs can be reassigned. A successful weight load only checks shapes. It does not prove that row 41 still means what it meant in the first 120,000 steps.

There are less obvious changes too. Normalization, pretokenization, special tokens, end of sequence markers and chat formatting can change which examples the optimizer sees. A different segmentation changes tokens per document, truncation, packing boundaries and the number of useful target tokens per step. If the learning rate schedule is driven by steps while the amount of text per step changes, the training recipe changes again. A smooth average loss can hide a regression in the affected language or a shift in which token positions are being predicted.

I would compare the immutable tokenizer files and settings from the checkpoint manifest with the exact preprocessing worker artifact. Encode a small fixed corpus spanning the affected language, control languages, tool and role markers, long documents and boundary cases. Compare token IDs, decoded text, sequence lengths and packed examples. Also inspect the data pipeline logs to identify the first mixed batch. A flag saying “tokenizer v2 deployed Tuesday” is not enough if workers rolled gradually or cached tokenized shards were mixed with fresh ones. The Hugging Face tokenizer documentation notes that adding tokens requires matching model embeddings. Equal matrix dimensions do not solve a changed mapping.

If the tokenizer was meant to remain fixed, roll the workers back and resume from the last checkpoint and data cursor before any changed examples entered an optimizer step. Replay or exclude the affected batches according to a recorded sample plan. Do not simply switch back at the next batch and pretend the mixed optimizer updates did not happen. If the change is intentional, treat it as a new training stage. Preserve old token IDs where possible, define any new embedding and output-head initialization, reset or transform affected optimizer state only with an explicit method, and reevaluate across languages and tasks. Record the lineage from the previous checkpoint rather than calling it the identical continuation.

What if the only change is faster tokenizer code and golden texts produce identical IDs? Then it may be a performance change, provided the full relevant input domain and packing output are equivalent. Test Unicode normalization, invalid bytes, special tokens and truncation too. I would still pin the artifact so the answer to this question is reproducible later. The model weights passed evaluation. Why did production tool calls break? asks why production serving used a tokenizer or chat template different from the evaluated weights. Here the training data itself changed midway, and the optimizer has already consumed some of it.