Why can a larger context window produce worse answers?

The first thing I would check is what else changed during the rollout. New model? New prompt order? More chunks? Different truncation rule? A larger available window does not hurt an answer by itself. The assembled input and the model using it are what changed.

There are several plausible failure paths. The right passage may be buried among near duplicates and contradictory versions. It may lose a qualifying paragraph because chunks were packed differently. A prompt that previously placed the rule near the question may now place it in the middle of a long input. The model may see a newer looking but less authoritative source and use that instead. Tool definitions, prior conversation, and reserved output tokens may also consume part of the budget, so the supposedly larger evidence set can still be trimmed in a different place.

I would inspect paired traces from old and new paths before proposing “put the good chunk last.” For the same question, corpus snapshot, authorization, model, and decoding setup where possible, record the exact assembled prompt and source versions. Compare the selected evidence, position of the required span, contradictory passages, and final claims. If the model changed with the context window, run the old and new context compositions on each model separately. Otherwise we cannot tell a context effect from a model effect.

Then run a small controlled grid:

“The relevant passage is always present” does not settle the issue. The answer may need two passages, or the cited passage may not include the exception. Presence also does not mean the model attended to and applied the right sentence. The Lost in the Middle study found position effects in its tested models and tasks. I would use it as a hypothesis, not as proof that every current model has the same failure curve.

If the experiment shows contradiction is the main issue, I would fix selection and authority metadata rather than rearrange token order. If position matters on the current model, keep the decisive evidence near the question and make the source boundary clear, while still including other necessary evidence. If clipped context is the problem, retrieve neighboring spans or a structured clause. None of these suggests that “shorter is always better.” A genuinely complex question can require more context.

The interviewer says the full context wins on rare questions that need three documents, though it loses on simple ones. I would not pick one global budget. Classify the evidence need from the question and retrieval results, or let a controlled second retrieval step fetch missing support. Compare the cost and wrong-answer rate of that policy against always-large and always-small baselines. A classifier error that drops a required third document can be worse than the original long prompt problem, so include that failure slice in the release gate.

I would call the rollout successful only when the measured answer and support quality improves in the relevant slices, not when token utilization or maximum window size rises. The extra capacity is useful when the system knows what to put in it.