The model consumes token IDs, not raw chat messages. The tokenizer maps bytes or text into those IDs, while the chat template turns roles, tool definitions and turns into the sequence the model was trained to understand. If production uses a different tokenizer vocabulary or template from evaluation, the same visible conversation can become a different input. A model loading without an exception proves very little about semantic compatibility.

I would compare a fixed set of conversations at the byte and token level between evaluation and production. Include system and user messages, a tool definition, assistant tool request, tool result, non-English text and special tokens. Record the rendered prompt, token IDs, decoded output and stop reason. Do the roles and tool boundaries match? Are tool calls parsed from the right output format? Was an end-of-turn token changed so the serving engine stops too early or leaks tool syntax into ordinary text? Hugging Face tokenizer documentation explains the tokenizer's role and warns that vocabulary changes must match model embeddings. vLLM serving documentation describes chat template requirements. Neither document says a given checkpoint and arbitrary template are interchangeable.

There are two distinct hazards. If token IDs or vocabulary changed, the weights may attach learned meaning to different IDs or lack trained embeddings for new tokens. If the vocabulary is identical but the chat template changed, the model may see roles and tool turns in a format it was not trained or tested on. Merely pinning the model name does not pin either. A gateway might also apply a second template or transform tools into a provider-specific schema. Trace the actual rendered input at the serving boundary with sensitive values protected.

The deployment artifact should identify immutable weight files or digest, tokenizer files and digest, chat template, model config, special-token map, generation defaults, tool parser, serving engine version and adapters. A release manifest ties them together. Build a compatibility test that compares golden token sequences across environments and a behavior test for tool use, multilingual prompts, long context and stop conditions. The token test catches packaging drift. The behavior test catches valid yet wrong configurations.

Do not repair this by changing temperature or hiding raw tool syntax in a postprocessor. Restore the evaluated bundle or perform a new evaluation on the intended bundle. If a tokenizer update really is needed, check the vocabulary mapping and embedding matrix, retrain or adapt where necessary, and roll it out as a model change. If the provider exposes only an opaque model identifier, record the strongest available version and use behavioral canaries because exact bytes may not be available.

If the interviewer says offline scores were good, ask what offline harness rendered. Perhaps the harness used the tokenizer bundled with the checkpoint, while production fetched main. Perhaps the evaluation never exercised tools or the affected language. The release gate should fail on any artifact mismatch and on task-level regressions by slice. This is different from an ordinary prompt edit. It changes the protocol the model actually reads.