Your LLM Rate Limiter Is Counting the Wrong Thing

Most LLM rate limiters start as normal API limiters.

Token bucket, requests per second, a sharded counter in Redis, some headers, maybe a dashboard. It works in the demo because demo traffic is clean and every request looks roughly the same.

Production traffic does not look like that.

One fact breaks the usual model: a request is not a unit of work.

A 50 token classification and a 200k token context summarization both count as one request. One barely touches the fleet. The other can burn prefill, create a large KV footprint, hold a stream open, push other requests into the tail, and make the serving engine behave differently. If the limiter treats both as one request, it is not protecting the serving system. It is decorating the API.

For an LLM API, rate limiting is really admission control. You are deciding whether the platform should accept work before that work consumes accelerator time, KV cache memory, scheduler slots, provider quota or customer budget. That decision is harder than it looks because the cost is multi dimensional, partly unknown at admission time, and changing with model, context length, cache state, traffic class and request mix.

This is the production design shape I would start from.

Start with the bottleneck#Copy link

A rate limiter protects a bottleneck. If you cannot name the bottleneck, the rest of the design is guesswork.

For a normal REST API, the request is often a tolerable proxy for work. Not perfect, but close enough. Maybe the bottleneck is database connections, CPU, thread pools, downstream QPS or some fixed backend capacity. You can count requests and get a useful control.

LLM serving does not give you that clean shape.

The backend resource is accelerator time and high bandwidth memory, and the work is split across two phases that stress the system differently.

Prefill processes the prompt before the first token appears. It is compute heavy, bursty and sensitive to prompt length. Long prompts are painful because attention work, memory movement and KV creation all happen before the user sees anything. The exact cost curve depends on the model, kernels and serving engine, but the operational point is simple: long prompts consume capacity very differently from short prompts.

Decode generates the response one token at a time. It is usually constrained by memory bandwidth, active sequences and KV cache capacity. Every sequence in flight holds KV cache proportional to its current context length. A streaming request is not just a request that passed a counter. It is an active claim on memory and scheduler slots for as long as the stream lives.

That split is the first thing to internalize. A million input tokens and a million output tokens do not hit the same wall. Input heavy traffic hurts prefill. Output heavy traffic holds decode capacity. Long context traffic does both, and it can change the tail latency for unrelated small requests behind it.

Continuous batching improves utilization, but it does not remove the ceiling. The serving engine is still deciding which sequences to admit, continue, preempt or finish. Paged KV cache helps use memory better, but if offered load pushes past what HBM and the scheduler can hold, the system falls into preemption, recompute or KV swapping. Throughput collapses, while a request count dashboard may still look normal.

There is also a newer serving shape where prefill and decode are disaggregated onto separate accelerator pools. That helps because the two phases have different resource profiles. Long prefill does not have to interfere directly with decode steps for already active streams. But it also makes admission more explicit: prefill heavy and decode heavy workloads are now separate capacity decisions, with separate queues and separate saturation points.

So the first rule is boring but important.

Protect the bottleneck you actually have, not the one the REST tutorial assumed.

Tokens are closer than requests, but raw tokens are still a proxy#Copy link

Requests per minute is not useless. Keep it as a cheap floor. It catches broken clients, tight retry loops, abuse and scheduler overhead from too many tiny requests.

But RPM cannot be the main control.

The primary unit is closer to tokens, and even tokens need to be split. Input tokens and output tokens are different because prefill and decode are different. A fresh input token and a cached prefix token are different. A token on a small model and a token on a large model are different. A short-context output token and a long context output token may not cost the same. Image, audio and tool heavy traffic are not the same resource class as plain text.

So a serious limiter needs a set of controls, not one bucket.

Requests per minute catches request count abuse. Input tokens per minute protects prefill. Output tokens per minute protects decode. Concurrency protects active KV cache, long lived streams and scheduler occupancy. Spend quota protects the billing boundary. Run budget protects agent workflows.

The run budget is easy to miss. A user sees one request, but an agent run may become five model calls, three tool calls, two repair prompts, one validation pass and a retry after a timeout. If you only limit each model call independently, the whole run can still explode while every individual call looks legal. The platform needs a budget for the logical workflow, not only the model calls inside it.

In practice, the limiter should think in resource classes or effective tokens. Raw tokens are what customers understand. Effective tokens are what the fleet feels.

A simplified internal model might look like this:

effective_cost =
  fresh_input_tokens * model_prefill_weight
+ cached_input_tokens * cached_prefill_weight
+ output_tokens * model_decode_weight
+ long_context_penalty
+ active_stream_cost

The exact weights depend on your stack. The formula is not the point. The point is that “one token” is still an approximation. Good systems expose raw token limits because customers need simple contracts, but internally they keep a more honest capacity model.

Admission has an unknown output problem#Copy link

Normal API rate limiting usually knows enough about the request when it arrives. LLM admission does not.

You can count or estimate input tokens before the model runs. You cannot know output length until generation finishes. max_tokens is only an upper bound, and sometimes the client does not set it. The model may stop after 50 tokens, run to the cap, stream slowly, call tools, retry, or fail halfway through.

So the limiter is asked to admit work whose final size is unknown.

The clean pattern is reserve and reconcile.

At admission, reserve the known input and some output budget:

reserved = input_tokens + reserved_output_tokens

The simplest version uses max_tokens as the output reservation. This is pessimistic and safe. It also wastes capacity because most requests do not use the full cap. Later, for stable routes, you can predict output length from historical distribution and reserve p90 or p95 plus margin.

But prediction is an optimization, not the baseline.

Prediction has a nasty failure mode. When many requests run long together, your prediction errors correlate exactly when the backend is already hot. You under reserve, over admit, and push the fleet into the thing the limiter was supposed to prevent. Reserve max is a good first build. Prediction should come later, only on routes with stable output distributions and enough headroom.

On completion, reconcile actual usage and refund the unused reservation:

refund = reserved_tokens - actual_tokens

The reservation must be a lease. Completion is not guaranteed. The client can disconnect, the worker can OOM, the stream can stall, the completion event can arrive twice or not at all. If reservations do not expire, tenants slowly lose usable quota to phantom requests, and the symptom looks like “we are being throttled below our limit.”

Every reservation needs a TTL. Long streams need heartbeat or lease extension. A reaper should refund orphaned leases. Completion reconcile must be idempotent on request ID so a duplicate event cannot double refund.

The same applies to concurrency slots. A concurrency cap is an active set problem, not a simple counter problem. If the system never observes completion, the slot leaks and the tenant eventually gets stuck.

At least once delivery is fine. Exactly once effect is what you need.

The admission path#Copy link

The limiter has to sit before expensive work starts. If prefill already happened, the limiter is late.

A useful request path looks roughly like this:

gateway
  -> authenticate org and key
  -> resolve model, route and traffic class
  -> apply cheap RPM guard
  -> build or estimate prompt shape
  -> count or conservatively estimate input tokens
  -> reserve input, output and concurrency
  -> check org, key, model and run budgets
  -> check spend boundary
  -> check backend pressure
  -> admit, degrade, queue or reject
  -> create reservation lease
  -> send to inference/backend
  -> reconcile on completion
  -> emit billing event from actual usage

RAG and tools make this path messier because the final prompt may not exist at the first gateway hop. The app may still need to retrieve chunks, rerank documents, add tool schemas, include conversation memory or expand a system prompt. In that case, use two gates.

The first gate is cheap: authenticate, RPM, rough size, run budget and abuse control. The second gate happens after context construction, when the system knows the real prompt shape. That is the actual capacity gate.

Use approximate token counting only for the cheap early gate. The final capacity decision should use exact tokenization where practical, or a deliberately conservative estimate with headroom. Approximation is a latency optimization, not permission to undercount.

Also handle missing max_tokens explicitly. If missing means “reserve the full model context window,” admission will seize on large context models. Use a sane default cap per route and model, and force the client to ask for more if it really needs more.

The controls are layered#Copy link

A production limiter is not one component. It is a stack of controls at different layers.

The edge layer protects tenant fairness and quota. It runs at the gateway, enforces org and key limits, holds reservations and rejects cheap. This layer should allow bursts within a tenant’s allowance because LLM traffic is naturally bursty.

The backend admission layer protects physical capacity. This is where queueing, deadline aware admission and concurrency ceilings belong. The accelerator fleet has a hard drain rate and finite active capacity. You cannot burst past it just because the edge token bucket allowed the tenant.

The scheduler layer inside the serving engine handles fine grained fairness between active sequences. Continuous batching decides which sequences make progress at each step. Edge limiting is coarse and early. Scheduler fairness is late and fine grained. You need both.

The trap is letting these layers fight. If the edge admits blind to real backend state, it can overshoot into scheduler thrash. If the backend rejects work after the edge already admitted it and after prefill has started, you burn expensive partial work and throw it away. That is how goodput collapses.

The fix is a feedback channel. The backend should publish pressure signals: KV utilization, prefill queue depth, decode queue depth, scheduler backlog, preemption rate, swap rate and deadline miss rate. The edge uses those signals to tighten or loosen admission.

But a feedback loop can become its own failure mode. Pressure rises, the edge clamps hard, load drops, the edge opens fully, pressure rises again. Now your limiter is oscillating. Use smoothing, hysteresis and rate limited adjustment. Use different thresholds for closing and opening. Do not wire raw backend pressure directly into a hard on/off gate unless you enjoy paging yourself.

Which algorithm belongs where#Copy link

The algorithm matters, but it is not the whole design.

Fixed window is the trap. It allows boundary bursts and creates a shared reset moment where throttled clients retry together. It is fine for a toy dashboard, not for a serious LLM API.

Sliding window log is exact but expensive because it stores one event per request. Sliding window counters are cheaper approximations and can work for simple request limits.

Token bucket is the right default for edge fairness. It gives a sustained refill rate plus burst capacity. That matters because interactive sessions, batch sweeps and agent runs arrive in bursts.

GCRA is a practical implementation of token bucket behavior. It stores one timestamp, usually the theoretical arrival time, and makes the decision with one atomic operation. It fits well in a central store. If you use it across gateways, use store time or another trusted time source. Gateway clock skew should not become a rate limit bug.

Leaky bucket and queue semantics fit backend capacity better. The backend has a physical drain rate and finite active work it can hold. Once it is full, you queue with deadlines, shed, degrade or reject.

The mistake is making one bucket do both jobs. Use token bucket or GCRA for tenant fairness at the edge. Use queueing, deadlines and concurrency for backend capacity. Use scheduler fairness for active sequences.

Multi tenancy and fairness#Copy link

The paid contract usually lives at the org level because the org pays. But one runaway API key inside that org should not starve the org’s production traffic.

So limits should be hierarchical.

The org bucket is the hard ceiling. Project, key or user buckets sit under it as fair share controls. Child buckets can be oversubscribed because not all keys are hot at once, but the parent keeps the aggregate honest.

The bucket key should include model or resource class:

(org, model, input_tokens)
(org, model, output_tokens)
(org, model, concurrency)
(org, model, spend)

A cheap model token and an expensive model token are not the same resource. If you collapse them into one org bucket, cheap traffic can hide expensive traffic and the limiter stops matching the fleet.

You also need committed and burst zones. Committed capacity is the tenant’s guarantee. Burst capacity is best effort. It can be used when the fleet has room, and it should be shed first when the fleet is under pressure. That is how you run the fleet hot without lying about guarantees.

Fairness under contention should be measured in work served, not request count. A tenant issuing many tiny requests and a tenant issuing a few long context requests are not equal. Request fairness is fake fairness here.

At the scheduler layer, the same idea shows up as weighted fair queueing, deficit round robin or virtual token style accounting. The exact algorithm depends on the serving engine, but the principle is stable: compare tenants by token denominated service received, not by number of requests admitted.

Noisy neighbor isolation also has a memory dimension. One tenant’s huge context can evict another tenant’s prompt cache or prefix cache, turning a cheap workload expensive. Strong isolation means per tenant cache budgets, per tenant cells or reserved capacity pools. That costs utilization but buys predictability.

Most systems choose a middle point: strict isolation for committed traffic, oversubscription for burst traffic, and enough telemetry to know when cache interference is hurting customers.

Behavior at the limit#Copy link

A weak limiter has one answer for every limit.

A production limiter reacts based on what tripped.

If spend is exhausted, reject with 429 and a Retry After pointing at the billing reset. Do not queue. Queuing money that does not exist is pointless.

If a tenant exceeds a transient rate limit on interactive traffic, reject fast with 429, Retry After, and honest limit, remaining and reset headers. Do not queue at the edge. Edge queueing hides backpressure and often creates latency the client will time out anyway. Push smoothing to the client through backoff with jitter.

Queue only where queueing is part of the contract: async jobs, batch jobs, offline processing.

If the backend is physically saturated, use a ladder.

Shed preemptible burst traffic first. Then degrade, but only if the request opted in. Degradation can mean smaller model, lower max_tokens, prompt cache path, reduced context, fewer reranked chunks or skipping optional expensive steps. Do not silently swap the model or change correctness behavior to save capacity. That keeps availability green while changing the product.

Then queue with deadlines. Admit to the queue only if the request can plausibly finish before its deadline. A queue bounded only by depth still accepts work that cannot complete in time, which wastes prefill later.

Then reject.

A fast clean 429 is more useful to a good client than a slow maybe. Reject is a feature.

Also separate tenant limit errors from fleet overload errors. A tenant limit error tells the client to slow its own rate. A fleet overload error tells the client the platform is hot and it should back off. If both look the same, SDKs cannot behave well.

Distributed enforcement#Copy link

The first build should usually be a central store with atomic scripts, sharded by tenant and model.

Redis with Lua, DynamoDB conditional writes, FoundationDB transactions, whatever fits your stack. The product matters less than the semantics: atomic reservation, lease creation, idempotent reconcile and low tail latency.

This first build is correct, easier to reason about and scales further than people assume.

It breaks for predictable reasons. The store’s p99 is now on every request path. A whale tenant creates a hot key. A store outage becomes an API outage. The problem is not “Redis is slow.” The problem is that a synchronous central decision sits in front of all useful work.

The scale design moves the hot decision local.

Each gateway holds a slice of tenant budget in memory and enforces locally. Admission is microseconds and has no network hop. A control plane redistributes slices every few hundred milliseconds. A durable ledger holds billing truth, but the request path does not read it.

Now the system has three timescales: hot path local budget, control plane allocation, and durable billing ledger.

The hard part is allocation, not counting.

Splitting a global limit as global / N fails as soon as traffic is skewed. Traffic is always skewed. Load balancers do not spread one tenant evenly. A tenant whose traffic lands on three nodes out of a hundred should not only get three percent of its limit.

The allocator needs demand weighted allocation. Nodes report recent per tenant demand. Idle nodes donate. Hot nodes borrow. The allocator pushes new slices. Usage merges asynchronously.

At very large scale, usage counters can merge through a central aggregator or gossip style CRDTs. A central aggregator is simpler to reason about. Gossip helps when the fleet is very large, multi region or when even the aggregator path becomes a bottleneck. What you should not put near admission is consensus.

Raft per request is not a rate limiter. It is a latency incident.

Sticky routing helps. Route a tenant to a bounded subset of nodes so its traffic lands near its budget slice. For normal tenants this keeps allocation simple. For whales, expand the subset or put them in their own cell.

Cells are often the highest leverage scaling move because they bound the problem. Route tenant to cell, enforce limits inside the cell, and keep capacity and blast radius local. A whale can get a dedicated cell. A limiter bug in one cell does not take down the fleet.

Multi region adds another split. A single global counter across regions is usually a no starter for latency. Allocate each tenant’s quota across regions based on demand, enforce locally, and reconcile globally. Accept that abrupt region shifts can briefly overshoot. Design the overshoot bound instead of pretending global strictness is free.

Local first enforcement always has an overshoot cost. If a tenant suddenly fans out across many gateways, every gateway may still hold some local burst allowance. Instantaneous over admission is roughly bounded by the number of gateways with stale allowance times their per node burst. That bound is tunable through slice size, refresh interval and burst size, but it is not zero.

For admission, approximate and fast usually wins. For billing, exact wins.

That distinction is the design.

The limiter gates. The meter bills.#Copy link

Do not use one counter for both protection and billing.

Rate limiting is protection. It can be approximate because it has to be fast. Over admitting by a small amount for a short time is usually acceptable if it keeps the hot path cheap.

Billing is money. It must be exact, and it can be asynchronous.

If admission waits for billing grade truth, the API becomes slow. If billing reads approximate limiter state, money becomes wrong.

Keep the paths separate. The limiter creates reservations and admission decisions. The worker emits completion records with actual usage. Billing reads completion events, not limiter counters.

A completion event should include request ID, org ID, model, input tokens, output tokens, cached tokens, status and timestamp. Aggregation should be idempotent on request ID because completion events will be delivered at least once, sometimes late and sometimes duplicated.

The limiter gates. The meter bills.

Two streams, not one counter used two ways.

Caching changes the price#Copy link

Caching breaks naive metering.

Prompt and prefix caching make long shared prefixes cheaper on prefill. If you charge every cached prefix token as full input TPM, you under utilize cache friendly workloads. If you charge it as zero, you over admit because the request still occupies KV cache and still decodes.

Meter effective cost.

Fresh input tokens pay full prefill cost. Cached prefix tokens pay less for prefill, but not zero for memory and active decode. Output tokens still pay decode cost.

The reservation should know the difference between cached and fresh input.

The dangerous part is that cache is shared and finite. Under memory pressure, cached prefixes get evicted. A workload that was cheap can become expensive with no warning, exactly when the fleet is already hot.

So cache aware metering is useful, but cache hits are not a hard guarantee. Treat them as capacity hints with failure modes.

Response caching is different. If a response is fully served from cache and never touches the model backend, it should not count against model capacity limits. It may still count against abuse controls, product quotas or cost policy. Capacity limiting and customer entitlement are related, but they are not the same meter.

The limiter also has its own caches: local budget slices, short negative decision caching and token count caches. Each one is a consistency tradeoff made to keep the hot path cheap. Keep TTLs short, and know what can go stale.

Failure modes that matter#Copy link

A production limiter earns its design in failure, not in the happy path.

Reservation leaks are the first failure. You reserve budget and concurrency at admission, but completion does not arrive. The client disconnects, the worker dies, the completion event is lost, or the stream stalls forever. Without TTL leases and idempotent reconcile, tenants get throttled below their real limits and nobody knows why.

Metastability is the fleet killer. Under saturation, admitting work, spending prefill and then dropping it later is wasted expensive work. Goodput falls below capacity and can stay there even after offered load drops. The defense is to shed early before prefill, use deadline aware admission and prefer finishing in flight work when hot.

Retry amplification is the partner failure. Clients retry 429s aggressively, or all retry at the same reset boundary, and a transient pressure event becomes sustained overload. Use continuous metering, jittered Retry After, retry budgets and SDK defaults that do not attack your own service.

Store failure needs a posture. Fail fully open and a limiter outage can flood a hot backend. Fail fully closed and the limiter becomes a single point of failure for a healthy backend. Fail rate limits to conservative local enforcement. Fail spend authority closedish near the boundary. Different state has different consistency requirements, so it should have different failure posture.

Slow consumers pin cache. A client that reads a stream slowly can hold KV cache and concurrency while making little progress. Use idle and stall timeouts distinct from normal generation timeout.

Hot keys happen when one tenant dominates one shard. Fix with tenant scaled sticky routing, sub allocation, whale cells or splitting the tenant’s bucket across replicas.

Output prediction has tail risk. If you reserve predicted output instead of max, many requests can run long together and over admit exactly when the fleet is hot. Use prediction only on stable routes with headroom.

Unset max_tokens can kill admission if it defaults to full context. Cap the default reservation.

Clock skew can break time based algorithms. Use store time where possible, monotonic clocks for elapsed time and NTP discipline.

Double refund happens when the reaper refunds an expired lease and a late completion refunds again. Make the reaper and completion path idempotent on the same request ID.

Sybil abuse bypasses per key limits by creating more keys. Put controls at an identity layer above the key: verified org, payment instrument, source network, free tier class. L7 rate limiting also does not replace L3 and L4 DDoS protection in front of it.

Feedback oscillation happens when backend pressure directly opens and closes edge admission without damping. Use hysteresis and gradual change.

These failures are not edge cases. They are the reason the design has leases, local fallbacks, deadlines, jitter, idempotency and separate billing.

The tradeoffs#Copy link

Most decisions in this limiter are not clean right or wrong choices. They are tradeoffs, and naming the tradeoff is usually more important than naming the algorithm.

Central enforcement is simpler and stricter, but it puts store latency on every request and creates hot key and availability risk. Local first enforcement is faster and more available, but it can over admit during slice staleness or tenant fan out.

Reserve max is safe, but it wastes capacity. Output prediction improves utilization, but it fails exactly when many requests run long together.

Reject fast is right for interactive traffic because the client can retry or degrade with full knowledge. Queueing is right for async and batch traffic because latency is not the product contract.

Strict tenant isolation gives predictability. Oversubscription gives utilization. The usual answer is committed capacity for guarantees and burst capacity that is explicitly preemptible.

Sticky routing keeps tenant traffic close to its allocation, but it can create hot subsets for whales. Cells contain blast radius, but they reduce pooling efficiency. Multi region local enforcement keeps latency low, but it accepts temporary global overshoot.

Fail open protects availability and risks backend collapse. Fail closed protects capacity and risks taking down healthy traffic. Local conservative is the middle path for rate limits. Spend should fail closedish near the boundary because money and capacity giveaways are harder to unwind.

A compact matrix looks like this:

The useful rule is this: enforcement can be approximate and fast because it protects capacity, while billing must be exact and async because it settles money.

Once that is clear, many design choices stop looking contradictory.

Scale paths and alternatives#Copy link

There are multiple ways to build this, and the right answer depends on where the system is in its life.

The first build should be central store enforcement with atomic GCRA and reservation leases. It is not the most impressive version, but it is the one you can reason about. It gives you correctness, visibility and a baseline.

Gateway or service mesh rate limiting can still be useful, but mostly for the RPM floor and abuse controls. Off the shelf gateway limiters count requests. They do not know about input/output tokens, output reservations, KV occupancy, prompt cache, run budgets or reconciliation. Buy the commodity edge protection. Build the token aware admission layer.

Local first allocation is the scale answer when the central store becomes a measured problem. Do not start there because it looks distributed. Start there when central hop p99 is eating the latency budget, hot tenants are pinning shards, or the store is too correlated a failure domain.

Cell based partitioning is the structural answer when tenants are large enough or blast radius matters enough. Put tenants into cells, give cells capacity pools, and keep most accounting local. This is often better than scaling one global limiter forever.

Multi region allocation is the geography answer. Split global quota into regional allocations, enforce locally, reconcile globally. Strong global rate enforcement across regions sounds clean, but it is usually too expensive in latency and availability.

Probabilistic structures like sketches can help for very high cardinality abuse signals, but tenant token accounting usually wants exact enough per tenant counters and leases, not a shared sketch that makes support disputes painful.

The migration discipline matters. Build the simple version first. Add complexity when metrics show the trigger. Otherwise you are building a design for the interview, not for the system.

Client contract#Copy link

The client is part of the limiter whether you like it or not.

If clients do not understand the limit, they discover it by hitting it. If SDKs retry badly, they become part of the overload.

Expose separate request and token limits in headers. Include remaining budget, reset time and Retry After. Tell the client whether it hit a tenant limit or fleet overload, because those require different behavior.

Support idempotency keys for ambiguous retries. A client retry should not create duplicate work or double charge.

Document backoff with jitter, and ship it in the SDK. Do not just put it in docs and hope customers implement it correctly.

Streaming needs its own contract. A request can pass admission, stream partial output and fail mid stream. The client must handle partial responses, and the server must bill and reconcile only what actually happened.

Usage tiers also matter. Raise limits as customers prove paid demand. Otherwise your support queue becomes the manual rate limit control plane.

Observability and operations#Copy link

You cannot tune a limiter you cannot see.

Emit a decision event for every request: allowed, throttled, shed, degraded, queued, rejected. Tag by org, model, route, limit type and traffic tier. Be careful with metric cardinality. Use aggregate metrics for dashboards and detailed events for drill down.

Dashboards should show the resource being protected, not only request rate. Show input tokens, output tokens, queue time, prefill pressure, decode pressure, KV occupancy, concurrency, cache hit rate, store latency, rejection reasons and retry behavior.

Run the limiter in shadow mode before enforcing new thresholds. Log what would have happened, then turn it on gradually.

Make limits dynamically configurable without a deploy, and version changes. A bad limit change is an outage. You need rollback.

Load test the limiter itself. Test central store p99, hot key behavior, local slice overshoot, completion delay, reaper correctness and store down posture. Run a game day where the store is slow, the store is down, one tenant fans out across every gateway and completion events arrive late.

The thing protecting the fleet from overload is also dangerous when it fails wrong.

What to build first#Copy link

Do not start with the most distributed version. That is usually complexity before evidence.

The first version should be boring and correct enough: RPM, input TPM, output TPM, concurrency cap, spend quota, agent run budget, reserve max with TTL leases, and a central store using atomic GCRA and reservation scripts sharded by tenant and model.

Separate the billing meter from the limiter on day one. That part is cheap to design early and painful to retrofit later. Run the limiter in shadow mode first, emit honest decision events, and expose clear headers so clients can back off correctly.

Then add complexity only when the trigger is real. Add backend pressure feedback when the edge is overshooting into scheduler thrash. Add local first allocation when central store p99, hot keys or store availability become real problems. Add cache aware metering when prompt cache hit rate changes capacity enough to matter. Add output prediction only when routes have stable output distributions. Add cells or multi region when blast radius or geography forces it.

The build order is part of the design. A simple limiter with the right units, leases, reconciliation and observability beats a clever distributed limiter nobody can operate.

The throughline#Copy link

An LLM rate limiter is not a counter in front of an API.

It is admission control for a system where work is measured in tokens, the final cost is not fully known at admission, the bottleneck moves between prefill, decode, KV cache, backend queues and billing, and one tenant can create load that looks small in requests and huge in capacity.

Count tokens, not just requests. Reserve before you know the final cost. Reconcile when you do. Keep enforcement fast and approximate. Keep billing exact and async. Protect the bottleneck you actually have, not the one the tutorial assumed.