The resource conflict is visible before we design a rate limiter. A 1,000 token prompt with a 100 token answer and a 60,000 token prompt with a 4,000 token answer are both one request. They do not use the same prefill work, KV memory, decode time, or provider tokens. If we meter only QPS, the large tenant can delay everyone while appearing well behaved.

I would account for at least three separate claims on capacity: input processing, live KV residency, and output generation. At the gateway, tokenize the input, inspect the requested output bound, and estimate an admission reservation. The output is not known in advance, so reconcile the reservation with observed usage while the stream runs. At the worker, apply per tenant shares to the resource that is actually scarce, including active KV tokens and decode time. The scheduler can let idle capacity be borrowed and reclaim it for tenants whose own work arrives. A fair LLM serving paper formalizes fairness using processed input and output work rather than request counts. Our exact weights still need to match our engine and product goals.

Fairness must have a waiting policy. If a tenant has already filled its share with long streams, admitting more of its requests into a global FIFO queue simply transfers the problem to everybody else. Bound queue work per tenant and class. Keep deadlines and expected start time visible, and reject or defer a request that cannot finish usefully. Existing long generations should not be interrupted casually. If the engine's preemption path recomputes a huge prefix, a policy that keeps evicting them can waste more capacity than it returns. Cap output length at an agreed boundary and give clients a way to continue deliberately if that is acceptable for the task.

Billing is related but not identical to scheduling. Bill for measured input and output tokens under the published price, including the provider's actual rules for cached input, failed attempts, and minimum charges. Also expose the cost drivers that tokens miss, such as reserved low latency capacity or a very long lived stream that occupies scarce KV. Whether those become explicit price components is a business decision. Do not invent a bill for GPU memory time after the fact. The meter should be versioned, auditable per request, and reconciled against provider usage and our own worker counters without retaining prompt contents unnecessarily.

What if the estimate was 500 output tokens and the model produces 4,000? The runtime needs a safe budget boundary. It can stop at the contracted limit, ask the application to request more, or extend within available tenant capacity. It cannot allow an unlimited run to bypass admission because tokens only became visible after the first chunk. Track cancellations too. A client that disconnects should release its reservation once the worker has actually stopped, not merely when the edge socket closes.

The tenant's work is now legitimately urgent. Urgency should be a purchased or authorized service class with a defined share and deadline, not a string in the request payload. Reserve capacity for it or use a separate pool if the burst is large. Permit borrowing from idle shares, then reclaim them without violating another tenant's guarantee. If every tenant can label every request urgent, we have recreated FIFO with a nicer name. Charge or limit the premium class according to the capacity promise and measure how much capacity is stranded when no urgent work arrives.

I would load test 200 tenants with mixed input and output lengths, bursts, cancellations, and a hot urgent tenant. Check per tenant queue age, TTFT, inter token gaps, admitted token work, preemptions, good completions, and invoice reconciliation. A fair design does not give every tenant the same completed request count. It gives them the contracted access to a scarce service without making another tenant's tail depend on a misleading QPS number.