One Tenant Can Break Your LLM Platform Without Sending Many Requests

The system design of multi-tenant LLM serving.

A tenant sends only 20 requests per minute.

On a normal API dashboard, that does not look dangerous. QPS is low. Request count is calm. Nothing screams overload.

But each request has a 150k-token prompt, asks for a long streamed answer, keeps the connection open, and runs inside an agent loop that retries when a tool call times out.

Now the platform is on fire.

This is the first trap in multi-tenant LLM serving.

The request is not the unit of load.

In a normal API, request count is often a useful starting point. Not perfect, but useful enough to reason about traffic, rate limits, and capacity.

In LLM serving, request count can be actively misleading.

One request can be 300 input tokens and 100 output tokens.

Another request can be 150,000 input tokens and 8,000 output tokens.

Both are one request.

They are not the same work.

That one difference changes almost every design decision.

Multi-tenant LLM serving is not normal SaaS multi-tenancy with GPUs attached. It is quota, scheduling, isolation, memory management, and cost control around uncertain token work.

That is the system.

---

The wrong starting point

The wrong starting point is simple.

Put the model behind an API. Add rate limits. Add autoscaling. Add billing. Done.

This works as a toy architecture. It does not survive production.

Normal web systems usually think in requests per second, CPU, memory, database QPS, connection count, and latency percentiles.

LLM serving needs all of that, but it also needs input tokens, output tokens, active KV cache, time to first token, inter-token latency, stream duration, batch occupancy, prompt-cache hit rate, agent fanout, and tenant-level fairness.

If your dashboard says QPS is low and users still see bad latency, this is usually why.

The system is not loaded by request count.

It is loaded by tokens, memory, and time.

---

Three facts that drive the design

There are many details in LLM serving, but most of the system comes from three facts.

1. Request cost is not fully known when the request arrives

When an LLM request arrives, you know the prompt size. You do not know the final output size.

You may know max_tokens, but that is only an upper bound. The model may stop after 50 tokens or generate 5,000 tokens.

So admission control is already dealing with uncertainty.

You can estimate. You can reserve. You can reconcile later. But you cannot know the real cost upfront.

The simple model does not work:

request arrives
cost is known
check quota
run request

The better model is:

request arrives
estimate cost
reserve capacity
run request
account while generating
finalize actual usage
refund or charge the difference

That is why quota and billing cannot be treated as the same thing.

Quota needs to be fast and approximate.

Billing needs to be exact and reconcilable.

2. Prefill and decode are different workloads

One LLM request has two phases.

First, prefill. The model reads the prompt. A long prompt means more work before the first token comes back. This decides TTFT, or time to first token.

Then, decode. The model generates one token, then another, then another. This decides streaming speed, or inter-token latency.

These two phases stress the GPU differently.

Prefill is more compute-heavy.

Decode is more memory-bandwidth-heavy.

That means one tenant can hurt everyone else in different ways. A long prompt hurts TTFT. A long output hurts streaming. Many active long-context sessions hurt memory.

So noisy neighbor in LLM serving is not one problem.

It is multiple problems hiding behind the same API.

3. GPU capacity is expensive and not instantly elastic

You cannot treat a large model serving fleet like a stateless web tier.

A stateless API server can scale fast. A large model replica is different.

You need the right GPUs, enough memory, loaded weights, warmed runtime, placement, parallelism, and enough batchable traffic to make the replica efficient.

Reactive autoscaling is useful, but it does not save you from a sudden spike by itself. By the time new capacity is fully useful, users may have already felt the spike.

This is why admission control, queue control, fairness, batch preemption, and headroom matter.

The system has to stay stable before new capacity arrives.

---

The actual thesis

The whole design comes down to this:

Multi-tenant LLM serving is a fairness and isolation problem on top of a memory management problem.

The scarce resource is not just GPU compute.

The scarce resources are prefill compute, decode bandwidth, KV cache memory, queue position, streaming slots, tenant budget, and sometimes tool-call capacity.

Tenants share all of this. One tenant should not be able to destroy another tenant’s latency. But if you isolate every tenant perfectly, utilization dies and cost explodes.

So the system is always trading isolation against utilization, latency against throughput, fairness against priority, cache hit rate against load balance, and cost against SLO.

That is the real design.

---

What one request does to the GPU

Before designing the distributed system, understand one request.

In prefill, the model processes the prompt. If the prompt has 100k tokens, the model has to process 100k tokens before generating the first answer token.

This is why long-context requests hurt TTFT. The user is waiting for the first token while the platform is doing prompt processing.

In decode, the model generates one token at a time. It cannot generate token 100 before token 99 exists, so decode is sequential per request.

Across requests, though, the server batches many active requests together.

That is the whole trick.

At each decode step, the GPU processes a batch of active sequences and produces one token for each sequence.

If batch size is 1, an expensive model step produces 1 token.

If batch size is 64, the same step helps produce 64 tokens.

This is why batching is not just an optimization.

It is the business model.

Without batching, serving large models is too expensive. With batching, many tenants share the same GPU step. That is where multi-tenancy becomes economic.

But batching has a limit.

The limit is memory.

---

KV cache is the real concurrency limit

Every active request stores attention state. That state is usually called KV cache.

It grows with the number of active tokens in the sequence. Prompt tokens count. Generated tokens count. Long conversations count. RAG context counts. Agent scratchpad tokens count.

This is why long-context serving is not just a bigger prompt problem. It is a memory residency problem.

The simple teaching formula for a dense MHA/GQA-style model is:

KV cache per token =
2 × layers × KV heads × head dimension × bytes

For a dense/GQA model shaped like Llama-3.1-70B, using 80 layers, 8 KV heads, 128 head dimension, and BF16/FP16 cache, the rough number is:

2 × 80 × 8 × 128 × 2
= 327,680 bytes
≈ 320 KB per token

That number is already enough to explain the first wall.

More active tokens means more KV memory. More KV memory means fewer concurrent requests. Fewer concurrent requests means smaller batches. Smaller batches mean worse throughput. Worse throughput means higher cost and worse latency.

But this formula is not the full story anymore.

It is the dense/GQA baseline.

Several recent large open models are designed specifically to fight this wall. GLM-5.2 is a good concrete example. As of June 2026, its public config exposes the serving-relevant numbers: 78 layers, 1M max context, MoE routing with 256 routed experts, 8 active experts per token, and compressed KV through kv_lora_rank = 512 plus qk_rope_head_dim = 64.

These numbers below are not benchmark numbers. They are just arithmetic on the published config.

For GLM-5.2, the useful MLA-style KV-cache calculation is closer to:

MLA KV per token =
(kv_lora_rank + qk_rope_head_dim) × bytes × layers

= (512 + 64) × 2 × 78
= 89,856 bytes
≈ 88 KB per token

Now compare that with a naive full per-head KV cache for the same model shape, without MLA compression:

naive full per-head KV per token =
2 × layers × KV heads × head dim × bytes

= 2 × 78 × 64 × 256 × 2
≈ 4.9 MB per token

Important detail: that 4.9 MB number is not a GQA cost. It is the naive full per-head MHA-style cost for the same shape.

The exact comparison I am using is:

That is the point.

The old formula is still useful because it teaches the wall. But current serving designs cannot blindly size memory using that formula. Modern long-context models are shaped around the wall.

MLA reduces KV memory.

MoE reduces active parameter bandwidth per token.

Sparse attention reduces long-context attention work.

This is why a much larger model can have a smaller KV footprint per token than an older dense/GQA model. The model architecture is not separate from serving. It is a response to serving physics.

For a multi-tenant platform, the conclusion does not change. KV cache is still the real concurrency limit. But the quota system must know the actual model’s KV profile, not just a generic formula copied from a previous generation model.

A tenant can be under request limits and even under token-per-minute limits, but if it holds many long-running, long-context streams, it can occupy KV memory and reduce concurrency for everyone else.

So KV cache must be treated as a schedulable resource.

Not an internal implementation detail.

The platform should track active KV tokens per tenant, active KV blocks per tenant, KV memory watermark per replica, cache dtype, model-specific KV bytes per token, and whether the model uses dense KV, GQA, MLA, sparse attention, or some hybrid.

If you do not track this, you get the weird incident where GPU compute is not full, QPS is not high, but TTFT is terrible.

That usually means memory or queueing, not raw compute.

---

Speculative decoding is one lever against the decode wall

Decode is painful because the model normally generates one token at a time. Inter-token latency is tied to the cost of each decode step.

Speculative decoding tries to reduce that pain by drafting possible future tokens and then verifying them with the target model. If the draft tokens are accepted, the system gets more than one useful output token from one verification step. If they are rejected, the extra draft work was wasted.

So speculative decoding is not magic.

It is a tradeoff.

It helps when acceptance length is high. It hurts when the draft path produces tokens that the main model rejects too often. In production, this becomes another runtime metric, not just a model feature.

You want to watch accepted tokens per step, rejected draft tokens, extra compute spent on speculation, and whether speculation is helping latency or only burning capacity.

This is also why newer model configs matter for serving. GLM-5.2 exposes an MTP layer for multi-token prediction, and its release notes talk about improving speculative decoding acceptance length.

That is not just a benchmark detail.

It is a serving detail.

If decode bandwidth is the wall, speculative decoding is one of the levers that can move it. But like every other lever in this system, it needs admission, scheduling, and observability around it.

---

Why request limits are not enough

Suppose Tenant A sends 20 requests per minute, each with 150k input tokens and 4k output tokens.

Tenant B sends 500 requests per minute, each with 300 input tokens and 100 output tokens.

Who is heavier?

By request count, Tenant B.

By actual GPU pain, probably Tenant A.

That is why LLM serving needs multi-dimensional quota.

At minimum, you need requests per minute, input tokens per minute, output tokens per minute, concurrent requests, max context length, max output tokens, active KV token budget, and daily or monthly spend.

For agents, add model calls per run, tool calls per run, parallel branches, wall-clock runtime, external API budget, and side-effect budget.

The quota should also be hierarchical:

organization
  workspace
    project
      api key or agent
        request

This matters because the noisy actor may not be the whole company. It may be one workspace, one app, one agent, or one API key stuck in a retry loop.

If quota only exists at org level, you either punish everyone or protect no one.

---

Admission control: reserve first, reconcile later

The request should not directly hit the GPU. It should first pass through admission.

The gateway authenticates the tenant, validates model access, checks context length, estimates input tokens, checks output limits, checks quota, checks spend, checks concurrency, and checks current overload state.

Input tokens are known.

Output tokens are not.

So one practical design is:

reserve input tokens
reserve predicted output tokens
cap by max_tokens
start the request
account output tokens while streaming
finalize actual usage at the end
refund unused reservation

Quota can be approximate. Billing cannot.

That split is important. Quota is a protection mechanism, so it has to be fast. Billing is a money mechanism, so it has to be correct.

The system should have two paths.

The quota path is near real-time, approximate, and allowed to be off by a small percentage. The billing path is append-only, idempotent, replayable, reconciled later, and based on engine token counts as source of truth.

If the gateway estimated 2,000 tokens but the engine generated 2,137 tokens, billing should use the engine count.

If the client retries, usage events should not double bill.

Use request IDs or idempotency keys.

This is not extra plumbing. This is the difference between a production billing system and a counter glued to an API gateway.

---

Architecture

The simple shape is:

The important split is this:

Gateway does coarse enforcement before admission.

Scheduler does fine-grained fairness while the request is running.

The gateway cannot solve everything because request cost is uncertain. The scheduler cannot solve everything because the system should not admit infinite work.

You need both.

The gateway protects the front door. The scheduler protects the GPU while work is running.

---

Routing is not just least-loaded

A weak router sends the request to the least-loaded GPU.

That sounds reasonable, but load is not one number.

A replica may have low QPS, high KV memory, low prefill queue, high decode pressure, good prompt cache for one tenant, and bad headroom for enterprise traffic.

The router should consider model requested, tenant tier, region or data residency, current queue depth, prefill backlog, decode pressure, KV memory watermark, prompt-cache affinity, tenant priority, and whether the request is batch or interactive.

Prompt cache creates a real tradeoff. If tenant requests reuse the same prefix, you want to route them to the same place to get cache hits. But if that replica becomes hot, cache affinity hurts load balance.

So the practical policy is simple.

Prefer the cache-affine replica unless it crosses a load threshold. Then overflow to a healthier replica.

Do not chase perfect global routing. The world changes every few milliseconds. Use coarse global routing and fine local scheduling.

---

FIFO is the wrong scheduler

A shared FIFO queue is the easiest way to build unfairness.

Tenant A sends 50 huge long-context requests. Tenant B sends small chat requests. FIFO puts B behind A, so B gets bad latency even though B did nothing wrong.

The scheduler should not schedule only requests.

It should schedule tenants.

A better shape is per-tenant queues. Tenant A has a queue, Tenant B has a queue, Tenant C has a queue, and the scheduler picks fairly across tenant queues.

Inside each tenant queue, you can pick by priority, deadline, size, or arrival time. Across tenants, use weighted fairness.

Enterprise gets more weight. Free tier gets less weight. But free tier should not starve forever, so you still need aging.

The exact algorithm can vary. The principle matters more.

Fairness must be based on consumed work, not request count.

For LLM serving, consumed work means input tokens, output tokens, active KV memory, and decode steps.

---

Why iteration-level scheduling matters

A shared FIFO queue is already unfair, but there is another problem below it.

The engine should not think only in whole requests.

During decode, the useful scheduling point is the iteration. One request may finish early. Another may keep generating for thousands of tokens. If batch membership only changes when the whole request finishes, short requests wait behind long generations and the GPU loses useful work.

Continuous batching fixes that by reshaping the batch at decode-step boundaries.

---

Separate prefill and decode thinking

Do not treat prefill and decode as the same queue. They create different user pain.

Prefill affects time to first token.

Decode affects streaming speed.

If long prefills dominate the GPU, active streams freeze. If decode dominates everything, new users wait too long before first token.

So the scheduler needs to reserve some capacity for decode, allow bounded prefill chunks, and interleave long prompt processing with active decode.

This is chunked prefill.

Instead of processing a 100k-token prompt in one huge block, split it:

prefill chunk
decode step
prefill chunk
decode step

Now the long-prompt tenant waits a bit more for first token, but everyone else’s streaming does not freeze.

That is usually the right tradeoff.

The chunk size is not just a performance knob. It is a fairness knob.

Large chunk gives better TTFT for the long prompt but worse ITL for others. Small chunk gives better streaming fairness but worse TTFT for the long prompt.

This is the type of knob an infra owner has to expose, tune, and monitor.

---

The noisy-neighbor taxonomy

Once you understand prefill, decode, and KV cache, noisy neighbors become easier to classify.

A design that only says “we will add a rate limiter” handles one row in this table.

Maybe.

A production system has to defend all of them.

---

Paged KV cache is not a small optimization

The KV cache should not be managed as one huge contiguous allocation per request. That wastes memory because most requests do not use their full max context. Some finish early, some grow slowly, and some share prefixes.

A better approach is to page KV cache into blocks, similar to virtual memory. Each sequence gets a logical view, while physical blocks are allocated as needed.

This gives less fragmentation, more active sequences, better batch size, easier preemption, prefix sharing, and copy-on-write for shared blocks.

This sounds low-level, but it changes the product.

More usable KV memory means more concurrency. More concurrency means better batching. Better batching means lower cost per token. Lower cost per token means better margins or lower price.

Memory management becomes product economics.

---

Prompt caching is not just a cache

Prompt caching saves prefill work when many requests share the same prefix. Common examples are system prompts, tool definitions, few-shot examples, long static documents, conversation prefix, and RAG boilerplate.

On a cache hit, the system does not need to reprocess the stable prefix. TTFT improves and prefill compute is saved.

But prompt cache is also a security boundary.

If Tenant A can observe that a prefix was cached because Tenant B used it, that is a leak. Even timing can leak information.

So the safe default is to scope cache by tenant, org, or workspace boundary.

That reduces hit rate, but it protects isolation.

Prompt cache creates three system problems.

Security problem: cache should not cross the tenant boundary promised by the product.

Routing problem: cache affinity can make hot replicas.

Fairness problem: a high-volume tenant can evict everyone else.

So you need tenant-scoped cache keys, cache-aware routing, per-tenant cache quota, and cache hit/miss metrics.

Cache hit rate is not a vanity metric.

It is capacity.

---

Batch traffic should be a shock absorber

Batch should not compete equally with interactive traffic.

Interactive traffic cares about TTFT, streaming speed, and low tail latency. Batch traffic cares about throughput, cost, and completion within a longer window.

So batch should run differently.

Online pools should be latency-sensitive and protected. Batch pools should be async, cheaper, preemptible, and allowed to use spare capacity.

Batch is useful because GPU traffic has peaks and troughs. During troughs, batch fills idle capacity. During spikes, batch gets paused or preempted.

This helps unit economics without hurting users.

The rule is simple:

Batch should harvest spare capacity, not steal the latency budget of interactive users.

For enterprise reserved capacity, batch can backfill idle reserved GPUs. But it must be preemptible quickly. If enterprise traffic returns, batch moves out.

That is how you sell isolation without wasting all idle hardware.

---

Agents make quota harder

Agent workloads are different from normal chat because one user request can become many model calls.

model call
tool call
model call
search
model call
code execution
model call
retry
model call
final answer

If quota is only checked per model call, the system misses the real unit.

The real unit is the run.

Agents need run-level budgets: max model calls, max tool calls, max runtime, max cost, max parallel branches, max retries, allowed tools, blocked tools, and approval policy.

Tool calls are also not equal.

A read-only search tool is usually safe to retry. Sending an email is a side effect and needs idempotency. Deploying a service is high risk and should require approval. Querying production data may need policy checks and audit.

So the serving system and the agent platform need to connect.

The model is not just generating text. It is driving actions.

If the platform does not put boundaries around that, the failure mode is not just high cost.

It is production damage.

---

Overload is where systems die

The dangerous overload pattern is not simple high load.

It is feedback.

One common loop:

This is how a system melts.

Not because one part failed, but because every part behaved locally reasonably.

The fix is to be strict earlier.

No unbounded queue. If projected wait time is too high, reject early. A fast 429 is better than a slow timeout.

Overload policy should protect paid interactive traffic, pause batch, shed free tier, cap low-priority max_tokens, reject over-quota tenants, return retry-after, and drop queued requests that already missed the deadline.

Do not silently downgrade answer quality.

If you route to a smaller model or reduce max output, the tenant should have opted in.

Silent quality degradation is not graceful degradation.

It is trust loss.

---

Failure modes worth designing for

Retry storm

A client times out and retries while the original request may still be running. Now the system does duplicate work. For normal text generation, this wastes GPU. For agents, this can duplicate side effects.

Fix it with idempotency keys, request deduplication, retry-after headers, SDK backoff, and per-tenant retry budgets.

Slow streaming client

The model is producing tokens, but the client reads slowly. Buffers grow, connection stays open, and resources stay pinned.

Fix it with stream buffer limits, disconnect detection, cancellation propagation, and idle timeouts.

Deploy cache stampede

A new model version rolls out and prompt cache gets invalidated. Suddenly every request becomes full prefill and TTFT spikes.

Fix it with staged rollout, cache pre-warm, old-replica draining, trough-hour deploys, and close monitoring of prefix-cache hit rate.

GPU dies mid-stream

In-flight KV is lost. For streaming, you cannot transparently resume because some tokens were already sent.

Fail the stream clearly, let the client retry with idempotency, and do not double bill. Non-streaming requests can retry internally if safe.

Metering pipeline lag

Quota counters become stale and billing events lag. The system should treat engine token counts as source of truth, usage events as append-only, and request IDs as idempotency keys.

A practical business rule: if the metering path is degraded, fail open for paid tiers and fail closed or stricter for free tiers.

Not everything is a pure technical decision.

---

Observability: QPS is not enough

If your dashboard only shows QPS, CPU, GPU utilization, and error rate, you are blind.

You need request metrics: TTFT, inter-token latency, end-to-end latency, queue wait, stream duration, cancellations, 429 rate, and 5xx rate.

You need token metrics: input tokens/sec, output tokens/sec, cached tokens/sec, cache creation tokens/sec, tokens per tenant, and tokens per model.

You need scheduler metrics: per-tenant queue depth, per-tenant wait time, fairness debt, active sequences, active KV tokens, preemption rate, batch size, prefill backlog, and decode pressure.

You need GPU and host metrics: GPU memory used, KV memory watermark, memory bandwidth pressure, batch occupancy, OOMs, host CPU tokenization pressure, detokenization latency, and streaming buffer pressure.

You need business metrics: cost per tenant, cost per completed task, cache savings, batch fill rate, and goodput by tier.

Goodput matters more than throughput.

Throughput includes work that missed the deadline and nobody used. Goodput is work completed within the promised SLO.

That is the number to protect.

---

Capacity planning is not “add GPUs”

Adding GPUs moves one wall. It does not remove all walls.

The walls usually show up in this order.

First, KV memory. This is often the first wall because it limits active sequences and batch size. When it saturates, TTFT rises, batch size drops, and GPU compute may still look underused.

Second, HBM bandwidth. This is the decode wall. When it saturates, inter-token latency rises and streaming feels slow. Increasing batch size helps throughput only until ITL breaks the SLO.

Third, prefill compute. This is usually spiky. A deploy, cache miss storm, or sudden long-context workload can make TTFT jump because prefill backlog grows faster than the system can drain it.

Fourth, control-plane hot keys. A large tenant can hit one quota shard too hard. Then gateway rate-check latency rises and admission slows down even before the GPU sees the request.

Fifth, interconnect. If prefill and decode are disaggregated, KV transfer becomes a real bottleneck. Prefill may finish, but decode cannot start because KV transfer is slow.

Sixth, host CPU. This is boring but real. Tokenization, detokenization, JSON handling, Python runtime overhead, and streaming can saturate CPU while GPU metrics look fine.

That is why capacity planning is not just GPU count.

It is choosing which wall you are moving and knowing which wall appears next.

---

Prefill/decode disaggregation

At some scale, chunked prefill may not be enough. Then you can split prefill and decode onto different pools.

Prefill pool handles prompt processing. Decode pool handles streaming generation.

Prefill workers compute KV. Decode workers consume it and stream tokens. This gives separate capacity knobs for TTFT and ITL.

That is powerful, but it adds cost and failure modes: KV transfer latency, interconnect pressure, two-pool scheduling, orphaned prefills, more complex debugging, and harder capacity planning.

So I would not start with it.

Start with a unified pool, continuous batching, chunked prefill, KV-aware scheduling, and good observability. Move to disaggregation when prefill interference is actually hurting p99 streaming latency and scale justifies the complexity.

Knowing when not to use a mechanism is part of the design.

---

Fine-tuned tenants and LoRA

Some tenants want custom model behavior. Running one full model replica per tenant is usually wasteful because traffic per fine-tune may be small.

A practical approach is to run a shared base model and load small adapters per request, while batching different adapters together if the runtime supports it.

This creates new serving problems: adapter cold start, adapter cache eviction, heterogeneous batches, per-adapter metering, and adapter isolation.

Same pattern again.

A product feature becomes a scheduling and memory problem.

---

Security isolation

Performance isolation is not enough. Tenants also need security isolation.

No cross-tenant leak through prompts, outputs, logs, traces, prompt cache, tool outputs, files, error messages, or batch execution.

Prompt cache is especially subtle. Even if content is not returned, timing can leak that a prefix was cached. So cache scope should be a tenant, org, or workspace boundary depending on the product contract.

For higher tiers, you may need zero data retention, region pinning, no prompt logging, disabled caching, audit trails, separate encryption keys, or dedicated capacity.

Every isolation feature has cost.

No logging hurts debugging. No caching hurts prefill cost. Region pinning strands capacity. Dedicated hardware hurts utilization.

That is why isolation has to be a priced product tier, not a hidden checkbox.

---

The tradeoff ledger

A serious design should say what it gives up.

This table is important because design is not listing mechanisms.

Design is choosing tradeoffs.

---

Phased build

I would not build the final system in one shot.

v1

Start with the system whose invariants are correct: shared model pool, continuous batching, paged KV cache, per-tenant queues, multi-dimensional quota, input/output token accounting, concurrency limits, basic prompt cache, idempotent usage events, and good observability.

This already gives a safe system.

v2

Make it more efficient: tenant-scoped prefix cache, cache-aware routing, better token prediction, KV quotas, preemption, batch tier, agent run budgets, and predictive autoscaling.

v3

Add advanced scale mechanisms: prefill/decode disaggregation, heterogeneous GPU pools, LoRA multiplexing, reserved capacity product, multi-region residency, and stronger enterprise isolation.

Do not start with v3.

Start with a system whose invariants are right, then deepen.

---

Invariants

These are the laws I would keep for the system. If one of these breaks, the design is wrong somewhere.

1. No unbounded queue. Overload becomes 429, not infinite latency.

2. Request admission must account for KV memory. Do not admit work that cannot fit or be reclaimed safely.

3. Quota is approximate and fast. Billing is exact and replayable.

4. Engine token counts are source of truth. Gateway counts are estimates.

5. Usage events are append-only and idempotent. Retry should not double bill.

6. Prompt cache never crosses the tenant boundary promised by the product. Hit rate is not worth data leakage.

7. Batch is preemptible. Otherwise it is not a shock absorber.

8. Higher-priority traffic should not wait behind lower-priority traffic at the same decision point.

9. Slow clients must not pin resources forever. Streaming needs backpressure and cancellation.

10. No silent quality downgrade. Brownout must be opt-in or explicit.

11. Goodput matters more than throughput. Work that misses deadline is waste.

These invariants make the system easier to review. During an incident, ask which invariant got violated.

---

One incident walkthrough

Suppose enterprise users report this:

p99 TTFT jumped from 1.2s to 12s
GPU utilization is only 55%
QPS is normal

A normal API mindset gets confused. GPU is not full. QPS is normal. So why is latency bad?

In LLM serving, I would check prefill queue wait, KV memory watermark, active KV tokens by tenant, batch size, prompt-cache hit rate, recent deploys, long-context traffic by tenant, batch jobs sharing the pool, retry rate, slow stream count, and router skew.

Likely causes: one tenant started long-context jobs, cache got invalidated after deploy, batch was not preempted, KV memory capped active sequences, or router overloaded cache-affine replicas.

This is the difference between seeing the system and seeing only the API.

QPS can be normal and the platform can still be overloaded because the overloaded resource is not QPS.

It is memory-time.

---

Final summary

Multi-tenant LLM serving is not request scheduling.

It is resource fairness under uncertain token cost.

The platform has to protect TTFT, streaming latency, GPU memory, tenant isolation, cost, billing correctness, and agent side effects while the request cost is not fully known upfront.

That is why the design needs two levels of control.

At the gateway: estimate, quota, reserve, admit or reject.

At the scheduler: fair queues, KV budgets, prefill/decode control, continuous batching, and preemption.

Everything else is a mechanism around that.

Prompt caching buys back prefill cost. Paged KV buys back memory. Batch buys back utilization. Chunked prefill buys back streaming fairness. Disaggregation buys separate TTFT and ITL knobs. Speculative decoding buys back decode latency when acceptance is good. Agent budgets prevent one request from becoming a runaway workflow.

The model gives intelligence.

The serving system gives guarantees.

And in a multi-tenant platform, the guarantees are what the tenant is actually paying for.

---

References

• Orca: A Distributed Serving System for Transformer-Based Generative Models
• vLLM / PagedAttention: Efficient Memory Management for Large Language Model Serving
• Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference
• DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized LLM Inference
• Fairness in Serving Large Language Models / Virtual Token Counter
• S-LoRA: Serving Thousands of Concurrent LoRA Adapters
• GLM-5.2 model card, release notes, and public config, accessed June 2026
• Llama-3.1-70B model card/config, for the dense/GQA comparison anchor