Model and Inference Engineering · Staff
The logits and seed stayed the same. Why did moving temperature before top-p change the answer?
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
Top-p is a decision about probability mass. Temperature changes that mass before we decide which tokens belong to the nucleus. Divide logits by temperature, normalize with softmax, then take the smallest ranked set whose cumulative probability reaches p. If we choose the set first and apply temperature only inside that set, a token excluded at the first step can never return. Those two pipelines define different sampling distributions. Hugging Face's generation configuration documents temperature and top-p as separate controls. The order is part of our sampler implementation, not something the seed can repair.
Take three logits 0, -1, -2. At temperature 1, their probabilities are about 0.665, 0.245, 0.090. With top-p at 0.8, we need the first two tokens. At temperature 0.5, the probabilities become about 0.867, 0.117, 0.016. The first token alone crosses 0.8. The rank order did not change because positive temperature preserves rank, but the membership of the nucleus did. That is why top-k and top-p react differently to temperature. Top-k membership stays the same in this simple case, while top-p's need not.
For a production refactor I would compare the full next-token probability distribution on small crafted logits, then use a seeded sampler as a secondary regression check. Include allowed-token masks, repetition penalties, minimum-token rules, padding tokens in a sharded vocabulary, ties and very low temperature. Write down the transform order once so CPU reference and GPU path implement the same distribution. A changed string under the same seed could be a harmless difference in random-number consumption, but a changed nucleus is an actual algorithm change. If the new order is intentional, treat it as a new API behavior. Users may have tuned temperature and top-p together, so compare quality and communicate the change.
Continue reading
Related questions
Read beyond the question
Explore more model and inference engineering
Follow another question in this area, or search the complete Question Library.
Browse this area →Browse Question Library →