The advertised window is an upper limit on a supported input shape, not a latency, concurrency, or answer quality promise. I would ask what product task actually needs a million tokens and what the result must be. A codebase investigation might need many files but only a small relevant subset at each step. A legal review may need coverage of an entire document and a way to show which parts were examined. Those are different requirements. Accepting a huge prompt because an API allows it can turn a retrieval problem into an expensive scheduling and evidence problem.

Start with memory. For a conventional decoder with full retained KV, a rough cache footprint is 2 × layers × KV heads × head dimension × bytes per element × live tokens. Use the actual architecture, not the parameter count, to fill that in. In the earlier illustrative 32 layer, eight KV head, 128 dimension, two byte example, that is 128 KiB per token. One million tokens would need about 122 GiB of KV before weights and runtime space. This is not a prediction for the proposed model. Grouped query attention, compressed KV, sliding windows, sparse attention, and distribution across devices change it. It shows why the input limit and the number of concurrent requests are inseparable.

The prefill work is also different from simply storing tokens. In ordinary full causal attention, the total interactions across a prompt grow roughly with the square of its length. Efficient attention kernels can avoid writing the full attention matrix to high bandwidth memory and improve wall time, but do not magically make exact full attention arithmetic linear. The FlashAttention paper makes that IO versus compute distinction. Actual models may use different attention structures, so benchmark this model's implementation. A million token prefill can occupy a worker long before the first visible answer token.

I would build a capacity sheet from measured prefill time, KV residency, decode time as the cache grows, transfer or offload cost if used, and p95/p99 queueing when a few huge requests mix with short ones. Admit large requests to a separate class or pool with explicit token and deadline budgets. Protect smaller interactive traffic. Consider context parallelism if the implementation can shard long input processing or KV, but it adds communication and needs a tested topology. The vLLM context parallel guide gives one concrete serving approach, not a promise that our model and hardware support it.

There is a quality gate independent of speed. Put the required fact at the beginning, middle, and end, then ask questions that need one span, two distant spans, a later exception, and an exact citation. Include irrelevant but plausible pages. Score supported claims and missed qualifiers, not just whether the model produced a confident summary. A model that handles a million input tokens syntactically may still miss a critical middle clause in the tasks we care about. If retrieval plus a smaller evidence set does better, that may be the right product path even when GPUs can carry the big input.

The probe says one request can consume most KV on a worker. Then admission cannot be “first come, first served” with a request count quota. Estimate live token occupancy from input plus reserved output, set a per-worker and per-tenant bound, and reject or queue a request whose required reservation would evict active work. The price and latency shown to the user need to reflect that class of service. If a customer really needs a full million token pass, we can offer a batch path with a different deadline and explicit evidence coverage.

I would ship the long window only for evaluated task slices and a load tested capacity class. A window advertised in a model card is a starting constraint, not the service design.