Rate limiting — the algorithms, and what they actually trade off

"Limit to N requests per window" sounds like one idea until you try to implement it — token bucket, leaky bucket, and fixed/sliding windows each make a different, deliberate trade-off.

Advanced

4 min read

What rate limiting is actually protecting against

A rate limiter rejects (or delays) requests once a client exceeds a defined threshold, protecting a service from being overwhelmed — by a genuine traffic spike, a buggy retry loop hammering an endpoint, or a deliberate abuse pattern like credential stuffing. The interesting part isn't the concept, it's that "N requests per window" has several genuinely different implementations, each with a different failure mode at the edges.

Fixed window — simple, with a real edge-case bug

Count requests in fixed, non-overlapping time buckets (e.g. "requests since :00 of this minute"), reset the counter to zero at each boundary:

class FixedWindowLimiter:
    def __init__(self, limit):
        self.limit = limit
        self.count = 0
        self.window_start = current_minute()
 
    def allow_request(self):
        if current_minute() != self.window_start:
            self.count = 0
            self.window_start = current_minute()
        if self.count >= self.limit:
            return False
        self.count += 1
        return True

The bug: a client can send LIMIT requests in the last second of one window, then immediately send another LIMIT requests in the first second of the next window — 2 * LIMIT requests in about one second, even though the configured limit is LIMIT per minute. The counter resets cleanly, but nothing prevents traffic from clustering right at the boundary.

Sliding window — fixes the boundary bug, costs more to track

Instead of a hard reset, a sliding window counts requests in the trailing N seconds relative to now, recomputed on every request — no fixed boundary for traffic to cluster around. A common approximation (sliding window counter) blends the current and previous fixed windows, weighted by how far into the current window you are, avoiding the cost of storing every individual request timestamp while still smoothing out the boundary spike.

Token bucket — the one that allows controlled bursts on purpose

A bucket holds up to capacity tokens. Tokens refill at a steady rate (e.g. 10/second). Every request consumes one token; if the bucket is empty, the request is rejected (or queued):

class TokenBucketLimiter:
    def __init__(self, capacity, refill_rate):
        self.capacity = capacity
        self.refill_rate = refill_rate       # tokens added per second
        self.tokens = capacity
        self.last_refill = now()
 
    def allow_request(self):
        elapsed = now() - self.last_refill
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
        self.last_refill = now()
        if self.tokens < 1:
            return False
        self.tokens -= 1
        return True

The key property: if a client has been quiet for a while, tokens accumulate up to capacity, so it can legitimately burst up to capacity requests instantly before being throttled back to the steady refill rate. This is a deliberate design choice, not a bug — it fits traffic that's naturally bursty (a user loading a page that fires several API calls at once) better than a hard per-second cap would.

Leaky bucket — the mirror image, smooths bursts instead of allowing them

Requests enter a fixed-size queue ("the bucket") and are processed ("leak out") at a strictly constant rate, regardless of how bursty the arrivals were. If the queue is full, new requests are dropped. Where token bucket says "save up capacity for a burst," leaky bucket says "no matter how bursty the input, the output rate is always smooth" — the right choice when what's downstream (a fixed-capacity worker, a third-party API with its own strict per-second cap) genuinely cannot handle bursts at all, even brief ones.

Where the limiter actually lives, and why it usually isn't per-server

Implementing the counter as a plain in-process variable only rate-limits requests that happen to land on that specific server — with N servers behind a load balancer, the same client could get roughly N times the intended limit by getting distributed across them. Real systems keep the counter in a shared store (commonly Redis, using INCR with a TTL, or a sorted set for sliding-window timestamps) that every server instance reads and writes against, so the limit is enforced against the client's total traffic, not their traffic to any one machine.

Algorithm comparison at a glance

Algorithm	Allows bursts?	Boundary bug?	Cost
Fixed window	✓ (at boundary)	✗ Yes	Low — one counter per window
Sliding window	✗ No	✓ No	Medium — weighted approximation
Token bucket	✓ Yes (intentional)	✓ No	Low — token count + timestamp
Leaky bucket	✗ No (smooths)	✓ No	Medium — queue management