The big idea
What is a rate limiter?
Without a guardrail, a single buggy client, scraper, or attacker can fire thousands of requests a second — exhausting your database, starving real users, and running up your bill.
A rate limiter caps how many requests a given caller may make in a window. Under the limit: pass through. Over it: reject immediately with HTTP 429, cheaply, before any real work happens. Simple in spirit — surprisingly subtle once it has to be fast, distributed, and exact.
How to read this: Each step opens with a real design decision — you make the call before I show you what ships. Watch the diagram grow, and at the end drop Redis to see the fail-open/closed dilemma live. Hit Begin.
Step 1 · The skeleton
A bouncer before the work
The expensive Upstream API shouldn’t waste a single cycle on a request that ought to be rejected. So the decision has to happen before it, as early as possible. Where?
Design decision: A request that should be rejected must not waste the expensive upstream. Where does the limit check go?
The call: At the API Gateway, before forwarding to the upstream. — Every request hits the limiter first: allow → forward, deny → 429 right there. The backend only ever sees approved traffic, so a flood never reaches the systems you’re protecting. Reject early, reject cheap.
Put the Rate Limiter at the API Gateway — the front door. Every request hits the limiter first. Allow → forward to the upstream service. Deny → return 429 Too Many Requests right there. The backend only ever sees approved traffic.
Reject early, reject cheap: The whole value is that a denied request costs almost nothing. Putting the check at the edge means a flood never reaches — and never overwhelms — the systems you’re protecting.
Step 2 · The rules
Limit what, for whom?
"100 requests" is meaningless without two answers: per what key (user? IP? API key?) and how much (free vs paid, per-endpoint).
Design decision: "100 requests" is meaningless alone. What two things must a limit specify?
The call: Per what key (user / IP / token) and how much (the limit). — The limiter builds a key — user:42, ip:1.2.3.4, key:abc:/search — and looks up that key’s limit from a Rule Store (free 100/min, paid 10k/min, strict endpoints stricter). The key decides what you’re actually protecting.
Add a Rule Store. The limiter identifies the caller and builds a key — user:42, ip:1.2.3.4, key:abc:/search — then looks up that key’s limit. Free tier 100/min, paid 10k/min, expensive endpoints stricter. Rules are cached in the limiter so the lookup isn’t itself a bottleneck.
Choosing the key: The limit key decides what you’re actually protecting. Per-IP stops anonymous floods but punishes users behind shared NAT; per-user/API-key is fairer but needs authentication first. Real systems layer several keys.
Step 3 · The algorithm
How do you count?
You need to track each key’s recent usage. A naive fixed window ("max 100 per minute") is easy but lets a caller fire 100 at 0:59 and 100 more at 1:00 — a 2× burst across the boundary.
Design decision: A fixed "100 per minute" window lets a caller fire 100 at 0:59 and 100 at 1:00 — a 2× burst. What counts better?
The call: A token bucket in Redis: refill at a steady rate, spend one per request. — Each key’s bucket refills steadily; a request spends a token, an empty bucket means deny. It allows healthy bursts while enforcing a long-run average — O(1) memory per key, µs per check. Sliding-window counters also fix the boundary.
Keep the counters in Redis (in-memory, microsecond ops). The favourite algorithm is the token bucket: each key has a bucket that refills at a steady rate; every request spends one token, and an empty bucket means deny. It allows healthy bursts while enforcing a long-run average. Sliding-window counters smooth out the boundary problem.
Token bucket vs windows: Token bucket: smooth average + controlled bursts (most common). Sliding-window log: exact but memory-heavy. Sliding-window counter: a cheap approximation that fixes the fixed-window burst. Pick the trade you can afford.
Step 4 · Make it distributed
Many gateways, one truth
You run dozens of gateway instances behind a load balancer. If each counts in its own memory, a caller gets N × limit by spreading requests across them — the limit is a fiction.
Design decision: Dozens of gateway instances each count in their own memory. What goes wrong, and how do you fix it?
The call: Centralize the count in shared Redis, incremented atomically. — Every instance reads/writes the same counter, and the read-modify-write is atomic (INCR with expiry, or a Lua script) so two instances incrementing at once can’t both slip under the limit. Shared + atomic = actually correct.
Centralize the count in shared Redis so every instance reads and writes the same counter. Crucially, do the read-modify-write atomically — INCR with expiry, or a small Lua script — so two instances incrementing at once can’t both slip under the limit (a classic race condition).
Atomicity beats races: Check-then-increment in two steps lets concurrent requests both see "99" and both pass. One atomic operation — increment and compare in a single round trip — is what makes a distributed limiter actually correct.
Step 5 · Go global
One Redis, or one per region?
The API now runs in three regions for latency. A caller in Tokyo hits the Tokyo gateway; a caller in Frankfurt hits Frankfurt’s. If each region has its own Redis, a caller who can reach multiple regions (or gets load-balanced across them) can rack up limit×regions in total throughput. If there’s one global Redis, every check crosses an ocean. Which do you pick?
Design decision: Gateways run in three regions. Where does the counter live?
The call: Regional Redis for the fast local check, with async replication/aggregation for the global total. — Each region enforces a local sub-limit instantly (fast, no cross-region hop), while a background process aggregates regional counts into a global view used to tighten local sub-limits over time. Trades perfect global exactness for the latency the whole system was built around.
Split the limit: each region’s Redis enforces a fast local sub-limit (say, limit/3 per region) with no cross-region hop, while an async process aggregates the regional counts and adjusts sub-limits if one region is idle and another is saturated. The global total is approximately right, not exactly right — the same recall-vs-latency trade every distributed counter makes at global scale.
Global correctness has a price: A perfectly exact global limit needs a synchronous global check, which reintroduces the latency multi-region deployment was built to avoid. Most systems accept approximate global enforcement (tight locally, eventually reconciled globally) rather than pay that cost on every request.
Step 6 · Fast & resilient
Don’t let the limiter fail you
Now every request makes a network hop to Redis — adding latency to the hot path — and if Redis hiccups, does your whole API go dark?
Design decision: Every request now hops to Redis, and if Redis hiccups your API could go dark. What do you do?
The call: Local token cache + a deliberate fail-open / fail-closed policy. — A small per-instance cache syncs with Redis periodically, trading a little precision for speed. And you choose up front: fail-open (allow if Redis is down — availability) or fail-closed (deny — protection). Public APIs usually fail open; abuse-sensitive paths fail closed.
Cut the hop with a small local token cache per instance that syncs with Redis periodically, trading a little precision for speed. And decide your failure mode up front: fail-open (allow if Redis is down — favour availability) or fail-closed (deny — favour protection). Most public APIs fail open; abuse-sensitive paths fail closed.
Fail-open vs fail-closed: When the limiter’s own state store is unreachable, you must choose: let traffic through and risk overload, or block traffic and risk an outage. There’s no free answer — pick deliberately, per endpoint.
Step 7 · See it working
Observe, signal, tune
A silent limiter is dangerous: you can’t tell a healthy deny from a misconfigured one strangling real users, and clients have no idea why they’re blocked.
Design decision: A silent limiter hides misconfigurations and leaves blocked clients confused. What do you add?
The call: Metrics (allow/deny, top offenders) + Retry-After / X-RateLimit headers. — Metrics let you tell a healthy deny from one strangling real users and spot attacks; standard headers turn a blunt 429 into a contract ("N left, retry in T") so good clients self-throttle — cutting load more than blocking does.
Emit metrics — allow/deny rates, top offenders — to spot attacks and tune limits. And be a good citizen to clients: return X-RateLimit-Limit, X-RateLimit-Remaining, and a Retry-After header on a 429 so well-behaved callers back off gracefully instead of retrying into the wall.
Cooperate with clients: Standard rate-limit headers turn a blunt 429 into a contract: "you have N left, try again in T seconds." Good clients self-throttle, which reduces the load far more than blocking ever does.
You did it
You just designed a rate limiter.
- Check at the gateway, before the work — reject early with 429.
- A rule store maps each key (user/IP/token) to its limit.
- Token bucket in Redis: smooth average plus controlled bursts.
- Shared, atomic counters make the limiter correct across instances.
- Local caching for speed; fail-open vs fail-closed for resilience.
- Metrics plus Retry-After / X-RateLimit headers to observe and cooperate.