The contract says 100 requests per second. Why did 200 requests pass at midnight?

Interview question. A public API promises to limit a client to 100 requests per second. The gateway uses a fixed one-second counter, yet clients can generate a burst almost twice the advertised rate around a window boundary. Compare fixed windows, sliding windows, token buckets and leaky buckets, and select an enforceable contract.

The dashboard says a customer submitted only one hundred requests in each second. The backend says it received two hundred almost simultaneously. Both observations can be correct. The limiter resets its counter at every wall-clock second, so a client sends one hundred at 12:00:00.990 and another hundred at 12:00:01.010. Each fixed window complies. The backend experiences two hundred arrivals over twenty milliseconds. Rate limiting has an averaging window, and the choice of window changes the meaning of the promise.

This is why I start by asking what the product is trying to protect. Are we charging for a monthly allowance, smoothing load on a database, slowing brute-force attempts, or allowing brief bursts for a legitimate interactive client? “100 RPS” is too incomplete to answer those questions. The rate, observation interval, permitted burst, scope of identity and enforcement error tolerance must be specified together.

A fixed-window counter is attractive because it is cheap. Keep a count for the current one-second bucket, increment atomically, and reject when it reaches one hundred. Only one counter value is needed per caller and window. The nasty edge effect is not an implementation bug. It is what the algorithm permits. It can be acceptable for approximate administrative quotas where a second-scale spike is harmless. It is much less useful as the only defense against a delicate downstream dependency.

A sliding-log limiter stores the timestamps of admitted requests for each identity and discards those older than the trailing window. For a request at 12:00:01.010, it sees the hundred admitted twenty milliseconds earlier and rejects the next wave. This enforces a precise rolling count, provided all decisions for that identity share an authoritative, atomic state. But if a popular tenant makes thousands of requests a second, maintaining each timestamp makes memory and update cost scale with traffic. Expiration work and coordinated clocks become part of the design.

A sliding-window counter is a compromise. Keep counts for the current and previous bucket, weigh the previous one by how much of it overlaps the moving window, and estimate usage. It is cheaper than a timestamp log and smoother than a fixed-window counter. It is still an approximation. If requests were concentrated at one edge of the previous bucket, the weighted estimate can be wrong. Saying “sliding window” without clarifying log versus weighted counter hides an important difference.

A token bucket answers a different question. It has a capacity B and a refill rate r. An accepted request consumes a token. When tokens are exhausted the request is denied or delayed, depending on the API contract. For capacity 150 and refill rate 100 tokens per second, a quiet client can build up 150 tokens and spend them immediately. After that, sustained traffic cannot exceed the refill rate over time. With a full bucket at time zero, the theoretical allowance over an interval of t seconds is bounded by approximately B + r × t, assuming atomic accounting and one token per request. This explicitly allows a burst rather than accidentally creating one at a window boundary.

One detail candidates frequently miss: a token bucket with a burst capacity of 150 does not protect a database that can never tolerate more than 100 concurrent heavy queries. Those are different resource quantities. Requests can consume varying CPU and query cost, and arrival rate is not equal to in-flight work. A capacity limiter may still be required behind the token bucket. Token charging can also be weighted by known operation cost, but an estimate should not be mistaken for a complete bound on execution work.

A leaky bucket is often described as a queue draining at a fixed pace. If two hundred requests arrive together, it releases them at a specified rate instead of passing the burst to the origin. That yields smoother downstream arrivals at the cost of queueing latency. A bounded queue must eventually reject when full. If the term “leaky bucket” refers instead to a continuously decaying counter, ask the interviewer which formulation is intended, because both usages appear in engineering literature. Do not promise smooth service from a counter alone.

The error contract matters as much as the algorithm. If the request is denied because a caller exceeded a rate policy, an HTTP 429 Too Many Requests response can identify the rule and optionally include Retry-After. That field may express a delay or an HTTP date. A client should not treat it as permission to synchronize thousands of retries at exactly the same instant. Jitter, retry limits and the user's own deadline still matter. A protected service may use 503 for its own overload, which is different from a customer-specific quota violation.

For a public API I would usually pick a token bucket to communicate steady rate and permitted burst clearly, possibly combined with a separate longer quota. For a security-sensitive fixed count, an exact rolling window may be worth its cost. For a worker with genuinely fixed processing capacity, a bounded paced queue can smooth work rather than reject bursts immediately. Then I would test an adversarial boundary burst, quiet-period burst, multi-identity fairness, clock drift, and a client reconnecting with many queued calls.

Interviewers are looking for more than four definitions. They want to see that the rate policy describes a temporal contract, that the implementation matches it, and that the chosen contract protects the real bottleneck rather than merely making the gateway graph look neat.