Your LLM Dashboard Is Lying About Load

The metric that runs your normal API can quietly mislead you on an LLM platform.

Picture a dashboard at 2am. QPS is flat, request count is low, error rate is clean, and by every signal a normal API trains you to trust, the system looks healthy.

It is not.

One tenant is sending 20 requests a minute. Each request carries a 150,000-token prompt, asks for a long streamed answer, keeps the connection open, and runs inside an agent loop that retries when a tool call times out. The graphs stay calm while the GPUs choke.

The dashboard is not broken. It is measuring the wrong thing.

That is the first thing to understand about multi-tenant LLM serving:

The request is not the unit of load.

In a normal API, request count is often useful. Not perfect, but useful enough to reason about traffic and capacity. In LLM serving, it can be actively misleading because one request can be 300 input tokens and 100 output tokens, while another can be 150,000 input tokens and 8,000 output tokens. Both count as one request. They are not the same work.

That one fact reshapes almost every design decision.

Multi-tenant LLM serving is not normal SaaS multi-tenancy with GPUs attached. It is quota, scheduling, isolation, memory management, and cost control around uncertain token work.

The wrong starting point

The usual product instinct is simple: put the model behind an API, add rate limits, add autoscaling, add billing, and call it a platform.

That works as a toy. It does not survive production.

Normal web systems think in requests per second, CPU, memory, and latency percentiles. LLM serving still needs those, but it also needs input tokens, output tokens, active KV cache, time to first token, inter-token latency, stream duration, batch occupancy, cache hit rate, and tenant-level fairness.

So when your dashboard says QPS is low and users still see bad latency, the system may not be contradicting the dashboard. The dashboard may be looking at the wrong unit.

The system is not loaded by request count. It is loaded by tokens, memory, and time.

Three facts drive the design

Most of the design falls out of three facts.

First, request cost is not fully known when the request arrives. You know the prompt size, but you do not know the output size. max_tokens is only a ceiling. The model may stop at 50 tokens or run to 5,000. Admission control is dealing with uncertainty from the first millisecond, so the system has to estimate, reserve, admit, and reconcile later.

Second, prefill and decode are different workloads. Prefill reads the prompt and decides time to first token. It is compute-heavy. Decode generates one token at a time and decides streaming speed. It is memory-bandwidth-heavy. This means one tenant can hurt others in more than one way. A long prompt hurts first-token latency. A long output hurts streaming. Many long sessions hurt memory.

Noisy neighbor is not one problem. It is several problems wearing the same uniform.

Third, GPU capacity is expensive and slow to add. A stateless API server can scale in seconds. A large model replica needs the right GPUs, loaded weights, a warm runtime, and enough batchable traffic to be efficient. Reactive autoscaling does not save you from a spike because by the time new capacity is warm, the spike is already felt.

The system has to stay stable before help arrives.

So the thesis is simple:

Multi-tenant LLM serving is a fairness and isolation problem on top of a memory management problem.

You are always trading isolation against utilization. Isolate every tenant perfectly and utilization dies. Cost explodes. Share everything and one tenant ruins the rest. The whole design lives in that tension.

KV cache is the real concurrency limit

Batching is the business model. At each decode step, the GPU produces one token for every active sequence. Batch size one produces one token from an expensive step. Batch size 64 produces 64 tokens from roughly the same step. Many tenants sharing one step is where serving becomes economic.

But batching has a ceiling, and the ceiling is memory.

Every active request stores attention state, the KV cache, and it grows with every token. Prompt tokens count. Generated tokens count. Conversation history counts. RAG context counts. Agent scratchpad counts.

All of it counts.

The teaching formula for a dense MHA/GQA-style model is:

2 × layers × KV heads × head dim × bytes

For a dense/GQA model shaped like Llama-3.1-70B, with 80 layers, 8 KV heads, 128 head dimension, and BF16/FP16 cache, that lands around 320 KB per token. That number is enough to explain the wall: more active tokens means more KV memory, more KV memory means fewer concurrent requests, fewer concurrent requests means smaller batches, and smaller batches mean worse throughput and higher cost.

But that formula is only the dense/GQA baseline, not the full story anymore. Many newer long-context models are built to fight this wall.

GLM-5.2 is a useful example because, as of June 2026, its public config exposes the numbers: 78 layers, MoE routing, 1M context, and compressed KV through kv_lora_rank = 512 plus qk_rope_head_dim = 64. Its MLA KV cache works out to about 88 KB per token. That is derived from the config, not benchmarked. The naive full per-head cost for the same shape would be about 4.9 MB per token.

Sit with that.

A far larger model holds roughly 3.6x less KV per token than the older 70B-style baseline because MLA was designed to defeat the memory wall.

The model architecture is not separate from serving. It is a response to serving physics.

For the platform, the lesson is still the same: KV cache is a schedulable resource, not an implementation detail. Track active KV tokens per tenant, KV watermark per replica, and model-specific KV bytes per token. A tenant can sit under every request and token-per-minute limit and still occupy KV memory with long sessions, dropping concurrency for everyone else.

When GPU compute looks idle, QPS looks normal, and latency is still terrible, the answer is usually memory or queueing.

Request limits are not enough

A tenant doing 20 requests of 150k tokens is heavier than a tenant doing 500 requests of 300 tokens, by every measure that matters.

So quota has to be multi-dimensional. You need request limits, input-token limits, output-token limits, concurrency limits, context-length limits, KV budgets, and spend limits.

Quota also has to be hierarchical: organization, workspace, project, API key, agent, request. The noisy actor is often not the whole org. It is one workspace, one app, one API key, or one agent in a retry loop.

If quota only exists at org level, you either punish everyone or protect no one.

Two control points

Enforcement happens at two altitudes.

The gateway does coarse work before admission: estimate cost, check quota, reserve, admit or reject. The scheduler does fine-grained work while the request is already running: per-tenant queues, weighted fairness based on consumed tokens, KV budgets, and chunked prefill so a long prompt does not freeze everyone else’s stream.

You need both.

The gateway cannot solve everything because cost is uncertain at admission. The scheduler cannot solve everything because the system should never admit infinite work. The gateway guards the front door. The scheduler protects the GPU while work runs.

Quota and billing are also not the same system. Quota is protection, so it must be fast and may be approximate. Billing is money, so it must be exact and reconcilable. Usage events should be append-only and idempotent, and engine token counts should be the source of truth.

That sounds boring until the first retry storm double-bills a customer or hides real usage.

FIFO is wrong here

A shared FIFO queue is the easiest way to build unfairness.

Tenant A sends 50 huge long-context requests. Tenant B sends small chat requests. FIFO puts B behind A, and B gets bad latency even though B did nothing wrong.

The scheduler should not schedule only requests. It should schedule tenants. Use per-tenant queues, weighted fairness, and work-based accounting. Give enterprise more weight. Give free tier less weight. But do not let lower tiers starve forever.

Fairness has to be based on consumed work, not request count. For LLM serving, consumed work means input tokens, output tokens, active KV memory, and decode steps.

There is another layer below this. The engine should not think only in whole requests either. During decode, the useful scheduling point is the iteration. One request may finish early, another may keep generating for thousands of tokens. If batch membership only changes when the whole request finishes, short requests wait behind long generations and the GPU loses useful work.

Continuous batching fixes that by reshaping the batch at decode-step boundaries.

Noisy neighbor has more than one shape

Once you understand prefill, decode, and KV cache, noisy neighbors become easier to see.

Huge prompts create prefill pressure and push up first-token latency. Long outputs create decode pressure and slow streaming. Many long sessions create KV pressure, which blocks new requests and drops batch size. Burst traffic creates queue pressure. Slow streaming clients pin resources. Prompt-cache thrash creates repeated prefill work. Agent loops multiply one user request into many model calls and tool calls.

A design that only says “add a rate limiter” handles one of these.

Maybe.

A production system has to defend all of them.

Agents make it worse

Agent workloads are different from normal chat. One user request can become a model call, tool call, model call, search, model call, code execution, retry, model call, and final answer.

If quota is only checked per model call, the system misses the real unit. The real unit is the run.

Agents need run-level budgets: max model calls, max tool calls, max runtime, max cost, max parallel branches, max retries, allowed tools, blocked tools, and approval policy.

Tool calls are also not equal. A read-only search is usually safe to retry. Sending an email is a side effect and needs idempotency. Deploying a service is high risk and should require approval.

So the serving system and the agent platform need to connect. The model is not just generating text. It is driving actions.

If the platform does not put boundaries around that, the failure mode is not just high cost. It is production damage.

Overload is where systems die

The dangerous failure is not high load. It is feedback.

KV memory fills, queues grow, time to first token rises, clients time out and retry, retries add fresh prefill work for answers nobody will read, queues grow more, and goodput collapses.

Nothing failed. Every part behaved locally reasonably. The loop killed the system.

The fix is to be strict earlier. No unbounded queue. If projected wait is too high, reject early. A fast 429 beats a slow timeout.

Under pressure, protect paid interactive traffic, pause batch, shed free tier, and drop requests that already missed their deadline. Do not silently route to a smaller model or cut output length. Silent quality downgrade is not graceful degradation. It is trust loss.

What to watch instead

QPS is not enough.

Watch KV watermark, queue wait, input tokens per tenant, output tokens per tenant, active KV tokens per tenant, prompt-cache hit rate, batch size, time to first token, inter-token latency, retry rate, and goodput by tier.

Goodput matters more than throughput. Throughput includes work that missed its deadline and nobody used. Goodput is work completed inside the promise.

That is the number to protect.

The point

Multi-tenant LLM serving is not request scheduling. It is resource fairness under uncertain token cost.

Estimate and reserve at the gateway. Run fair queues and KV budgets at the scheduler. Everything else is a mechanism around that.

Prompt caching buys back prefill. Paged KV buys back memory. Batch buys back utilization. Chunked prefill buys back streaming fairness. Speculative decoding buys back decode latency when acceptance is good. Agent budgets prevent one request from becoming a runaway workflow.

Request count and QPS will tell you the platform is fine right up until it is not.

The work the model does is intelligence. The work the system does is keeping that intelligence fair, bounded, and paid for.

On a shared platform, that second job is the product.