I would look at the mask alignment before blaming the cache contents. In prefill, query and key lengths often match, so a top-left triangular mask and a mask aligned to the last key positions can look the same. In incremental decode, we might have one new query and 1,000 keys that include earlier tokens. That query is position 999, not position zero. A local window must be placed around its actual position in the full sequence.

Take a tiny case with eight keys and two new queries. Those queries correspond to positions 6 and 7. With three prior positions in the left window and no right window, query 6 may read keys 3 through 6, and query 7 may read keys 4 through 7. If our wrapper builds a two-row mask as if the queries were positions 0 and 1, it points at the oldest keys, or masks every valid key. The shapes still pass and the output can still be finite. The FlashAttention interface documents bottom-right alignment when query and key lengths differ and defines its local window relative to both lengths. Check the contract of the particular kernel version in use.

I would write the expected allowed-key indices by hand for sequence lengths 1, 2 and 8, then compare a simple reference attention implementation with the optimized path. Include one-token decode, chunked decode, left padding, different cache lengths in one batch and the boundary where an old key leaves the window. Compare masks first, then attention outputs and logits. A prefill-only parity test misses the exact shape that fails.

If the interviewer says the sliding cache already evicted old keys, that changes the available key range but does not excuse a wrong alignment over the keys that remain. Ask whether the kernel sees absolute positions, compacted cache positions or both. Do not apply an offset twice. The sliding-window KV cache fits. Why did evicting every layer break long prompts? asks which layers are allowed to evict history. This question assumes local attention is intended and tests whether the query reads the right local keys at decode time.