Rate Limiting

A service can only do so much work per second. Push more at it and latency climbs, queues grow, memory swells, and eventually it falls over — taking everyone’s requests down, not just the excess. Rate limiting is the deliberate act of saying “no” to some requests so the service can keep saying “yes” to the rest. It’s a throttle: it caps the rate of accepted work to a level the system can actually sustain.

The motivations are twofold and worth separating, because they pull in slightly different directions. Protection: stop any source — a buggy client in a retry loop, a scraper, a denial-of-service attack, a sudden viral spike — from overwhelming the service. Fairness: ensure no single user can monopolize shared capacity at the expense of everyone else. The same mechanism serves both, but “protect the service” and “be fair to users” occasionally want different limits, as we’ll see.

The core algorithms

There are four classic algorithms. They differ in how they handle bursts — short spikes of traffic above the average rate — which turns out to be the whole game.

Token bucket — allow controlled bursts

Imagine a bucket that holds up to B tokens and refills at R tokens per second. Each request must take one token to proceed; if the bucket is empty, the request is rejected (or queued). Because the bucket can hold up to B tokens, a client that’s been quiet can spend a burst of up to B requests at once, then is limited to the steady refill rate R thereafter.

   refill at R tokens/sec
            │
            ▼
        ┌───────┐  bucket holds up to B tokens
        │ ● ● ● │  each request removes one token
        │ ● ●   │  empty bucket → request rejected
        └───────┘

This is the most widely used algorithm because the burst behavior matches reality: real users are bursty (a page load fires ten requests at once), and token bucket accommodates legitimate bursts while still enforcing a long-run average. Buys: flexibility for natural bursts. Costs: a client can still hit you with a full bucket’s burst, so downstream must tolerate B at once.

Leaky bucket — smooth the output

A leaky bucket is a queue that drains at a fixed rate R. Requests pour in and are processed at a steady drip regardless of how spiky the arrivals are; if the queue fills, new requests overflow and are dropped. Where token bucket allows bursts through, leaky bucket absorbs them and emits a perfectly smooth stream.

Buys: a constant, predictable output rate — ideal when the thing you’re protecting hates bursts (e.g., a downstream with a hard per-second ceiling). Costs: added latency (requests wait in the queue) and no burst friendliness — a legitimate spike gets smeared out or dropped.

Fixed window — simple, but bursty at the seams

Count requests per fixed time window: “max 100 per minute.” Reset the counter each minute. Dead simple and cheap. The flaw is the boundary burst: a client can send 100 requests at 11:00:59 and another 100 at 11:01:00 — 200 requests in one second, double the intended limit, because the window reset between them.

   minute window      |  minute window
   ......... 100 reqs  | 100 reqs .........
            ^ 11:00:59 | 11:01:00 ^
            └─ 200 requests in ~1 second ─┘

Sliding window — accurate, at a cost

Sliding-window approaches fix the boundary problem by considering a rolling time range rather than fixed buckets. Sliding window log keeps a timestamp for every request and counts those within the last 60 seconds — exact, but memory-hungry (you store every request). Sliding window counter approximates this by weighting the previous window’s count, getting most of the accuracy with far less storage.

Algorithm	Bursts	Memory	Accuracy
Token bucket	Allows up to B	Low	Good
Leaky bucket	Smooths away	Low (queue)	Good
Fixed window	Boundary spikes	Lowest	Poor at edges
Sliding window	Controlled	Higher	Best

Where to enforce it

Rate limiting is most effective at the edge, before bad traffic consumes deep resources. The earlier you reject a request, the less it costs you. The natural home is the API gateway or reverse proxy — it sees all inbound traffic, can limit per-API-key/IP/user before requests ever reach your services, and centralizes the policy. You’ll often layer limits: a global limit at the edge (protect the whole system), per-tenant limits (fairness), and a per-service limit deeper in (protect one component).

Distributed rate limiting is harder than it looks

With one server, the counter lives in memory. With a fleet behind a load balancer, every node must share a consistent view of “how many requests has this user made,” or each node enforces its own limit and the effective limit is N × your intended value. The common fix is a shared fast store (e.g., Redis) holding the counters — which adds a network hop and a new dependency to every request. Building this end-to-end is its own design problem, worked through in Design a Rate Limiter.

Protection vs. fairness, and what to do at the limit

The two motivations can conflict. A strict fairness limit (each user gets 1/N of capacity) can leave the service underused when most users are idle — protection alone would happily let one active user use the spare capacity. A pure protection limit (cap total load) can let one heavy user starve everyone else right up until the global ceiling. Real systems blend them: a generous per-user limit for fairness, plus a global limit for protection, and sometimes dynamic limits that tighten as the system approaches saturation.

When a request exceeds the limit, you have choices, each a trade-off. Reject it (return HTTP 429 “Too Many Requests”, ideally with a Retry-After header so well-behaved clients back off) — fast and protective, but the client loses the request. Throttle/queue it — kinder to the client, but adds latency and risks the queue itself becoming a resource sink. The honest signal is the 429 with Retry-After: it tells clients to slow down rather than retry-storm you, tying back to the backoff discipline from Timeouts, Retries & Backoff.

What does rate limiting buy us, and what does it cost? It buys survival under overload and protection against abusive or runaway clients — a service that throttles excess stays up for everyone. It costs you rejected legitimate requests at the margin, the operational burden of choosing and tuning limits, and (at scale) a shared-counter dependency on the hot path of every request.

By the numbers — a token bucket’s true peak rate

Take a token bucket sized B = 40 that refills at R = 20 tokens/sec, and a client that’s been idle long enough that the bucket is full. In the very first second it can spend all 40 stored tokens plus the 20 that refill during that second — 60 requests — then settles to a steady 20/sec:

second 1:   40 (burst)  +  20 (refill)  =  60 requests
second 2+:   0          +  20 (refill)  =  20 requests/sec

So the worst-case first second is B + R, not the advertised R. The downstream you’re protecting has to survive that peak, which is the real meaning of “downstream must tolerate B at once.” It’s also why a service that advertises “20 requests/second” almost always means a refill rate with a separate burst allowance — size the bucket from what the thing behind it can actually absorb, not from the average you wish for.

The architect’s lens

Before you wire a limiter into the hot path, answer the five questions:

• Why does it exist? Because a service can only do so much work per second, and unbounded demand — a buggy retry loop, a scraper, a DoS, a viral spike — pushes latency and queues past the point of collapse, taking everyone’s requests down, not just the excess.
• What problem does it solve? Two at once: protection (no single source can overwhelm the service) and fairness (no single user can monopolize shared capacity). The throttle caps accepted work at a level the system can actually sustain.
• What are the trade-offs? Each algorithm trades burst behavior against cost: token bucket allows controlled bursts up to B (so downstream must survive a B + R peak); leaky bucket smooths to a steady R but adds queue latency; fixed window is cheapest but leaks 2× at the boundary; sliding window is most accurate but memory-hungry. And protection vs. fairness occasionally want different limits.
• When should I avoid it? When the work is genuinely elastic and you’d rather autoscale than reject; when a strict fairness cap would leave the service idle while most users are quiet; or when the shared-counter dependency (Redis on every request) costs more than the overload it prevents.
• What breaks if I remove it? A single runaway or abusive client saturates the service for everyone, and without enforcement at the edge/gateway that bad traffic consumes deep resources before you can reject it. Across a fleet, naive per-node counters give an effective limit of N × what you intended.

Check your understanding

1. Contrast token bucket and leaky bucket specifically in how they handle bursts. When would you prefer each?
2. Explain the fixed-window boundary problem with a concrete example, and how sliding window fixes it (and at what cost).
3. Why is the edge / API gateway the natural place to enforce rate limits? What does layering limits at multiple tiers buy you?
4. Why is rate limiting across a fleet of servers harder than on a single server, and what’s the standard fix and its downside?
5. Distinguish the “protection” and “fairness” motivations, and give a scenario where a limit serving one undermines the other.

Show answers

1. Token bucket allows bursts: a quiet client can spend up to B tokens at once, then is limited to the refill rate R — it accommodates natural bursts (a page load firing ten requests). Leaky bucket smooths bursts away: it drains at a fixed rate R and emits a perfectly steady stream, queueing or dropping the excess. Prefer token bucket when bursts are legitimate and downstream can tolerate them; prefer leaky bucket when the thing you’re protecting hates bursts and needs a constant output rate.
2. With a fixed “100 per minute” window, a client can send 100 requests at 11:00:59 and another 100 at 11:01:00 — 200 in ~1 second, double the intended limit, because the counter reset between them. Sliding window fixes this by considering a rolling time range rather than fixed buckets; the cost is more memory (a log stores every request timestamp) or some approximation error (a sliding window counter weights the previous window).
3. The earlier you reject a request, the less it costs you — the edge / API gateway sees all inbound traffic and can reject before requests reach deep resources, while centralizing the policy. Layering limits buys defense at multiple tiers: a global limit at the edge (protect the whole system), per-tenant limits (fairness), and a per-service limit deeper in (protect one component).
4. With one server the counter lives in memory; with a fleet behind a load balancer, each node enforces its own counter, so the effective limit becomes N × your intended value unless nodes share a consistent view. The standard fix is a shared fast store (e.g., Redis) holding the counters — the downside is a network hop and a new dependency on the hot path of every request.
5. Protection caps total load to keep the service up; fairness ensures no single user monopolizes shared capacity. They conflict: a strict fairness limit (each user gets 1/N) can leave the service underused when most users are idle, where protection alone would happily let one active user absorb the spare capacity — and conversely a pure protection limit lets one heavy user starve everyone up to the global ceiling.