What is a rate limiter?

A rate limiter caps how many requests a given caller may make in a window. Under the limit: pass through. Over it: reject immediately with HTTP 429, cheaply, before any real work happens. Simple in spirit — surprisingly subtle once it has to be fast, distributed, and exact.

A bouncer before the work

Put the Rate Limiter at the API Gateway — the front door. Every request hits the limiter first. Allow → forward to the upstream service. Deny → return 429 Too Many Requests right there. The backend only ever sees approved traffic.

Limit what, for whom?

Add a Rule Store. The limiter identifies the caller and builds a key — user:42, ip:1.2.3.4, key:abc:/search — then looks up that key’s limit. Free tier 100/min, paid 10k/min, expensive endpoints stricter. Rules are cached in the limiter so the lookup isn’t itself a bottleneck.

Keep the counters in Redis (in-memory, microsecond ops). The favourite algorithm is the token bucket: each key has a bucket that refills at a steady rate; every request spends one token, and an empty bucket means deny. It allows healthy bursts while enforcing a long-run average. Sliding-window counters smooth out the boundary problem.

Many gateways, one truth

Centralize the count in shared Redis so every instance reads and writes the same counter. Crucially, do the read-modify-write atomically — INCR with expiry, or a small Lua script — so two instances incrementing at once can’t both slip under the limit (a classic race condition).

Don’t let the limiter fail you

Cut the hop with a small local token cache per instance that syncs with Redis periodically, trading a little precision for speed. And decide your failure mode up front: fail-open (allow if Redis is down — favour availability) or fail-closed (deny — favour protection). Most public APIs fail open; abuse-sensitive paths fail closed.

Observe, signal, tune

Emit metrics — allow/deny rates, top offenders — to spot attacks and tune limits. And be a good citizen to clients: return X-RateLimit-Limit, X-RateLimit-Remaining, and a Retry-After header on a 429 so well-behaved callers back off gracefully instead of retrying into the wall.

Design a Rate Limiter — A Guided System Design

System Design · step by stepDesign a Rate Limiter

Step 1 / 9

RUN IT YOURSELF

The token bucket, in Python & TypeScript

The most common rate limiter is a token bucket. Here it is in both languages, running live in your browser. Switch tabs, read the comments, edit the capacity/refill, and hit Run.

HOW TO READ THE CODE — 4 IDEAS

A bucket holds up to capacity tokens and refills at a steady rate.
Before each check, add tokens for the time elapsed since last time (step 1).
A request spends one token; if one is available it passes (step 2).
An empty bucket rejects the request — that is the rate limit biting (step 3). Capacity sets the burst size.

CPython · WebAssembly

# Rate limiter (token bucket): a bucket refills at a steady rate and
# holds at most "capacity" tokens. Each request spends one token; an
# empty bucket means the request is rejected.

class TokenBucket:
    def __init__(self, capacity, refill_per_sec):
        self.capacity = capacity
        self.refill = refill_per_sec
        self.tokens = capacity      # start full
        self.last = 0.0             # time of the last refill (seconds)

def allow(self, now):
        # STEP 1 - add tokens for the time elapsed since last check.
        elapsed = now - self.last
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill)
        self.last = now
        # STEP 2 - spend a token if one is available.
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False                # STEP 3 - empty bucket -> reject

bucket = TokenBucket(capacity=3, refill_per_sec=1)
# 5 requests at t=0: first 3 pass (burst), last 2 are rejected.
print([bucket.allow(0) for _ in range(5)])

// Rate limiter (token bucket): a bucket refills at a steady rate and
// holds at most "capacity" tokens. Each request spends one token; an
// empty bucket means the request is rejected.
class TokenBucket {
  capacity: number;
  refillPerSec: number;
  tokens: number;
  last = 0;

constructor(capacity: number, refillPerSec: number) {
    this.capacity = capacity;
    this.refillPerSec = refillPerSec;
    this.tokens = capacity;         // start full
  }

allow(now: number): boolean {
    // STEP 1 - add tokens for the time elapsed since last check.
    const elapsed = now - this.last;
    this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.refillPerSec);
    this.last = now;
    // STEP 2 - spend a token if one is available.
    if (this.tokens >= 1) {
      this.tokens -= 1;
      return true;
    }
    return false;                   // STEP 3 - empty bucket -> reject
  }
}

const bucket = new TokenBucket(3, 1);
// 5 requests at t=0: first 3 pass (burst), last 2 are rejected.
console.log(Array.from({ length: 5 }, () => bucket.allow(0)));

Finished this one? 0 / 28 System Designs done