The whole design, in writing
Learn system design by building an API rate limiter step by step. An interactive guide covering edge enforcement, limit keys and rule tiers, the token-bucket algorithm, distributed atomic counters in Redis, local caching, fail-open vs fail-closed resilience, and rate-limit headers.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a rate limiter?
Without a guardrail, a single buggy client, scraper, or attacker can fire thousands of requests a second — exhausting your database, starving real users, and running up your bill.
A rate limiter caps how many requests a given caller may make in a window. Under the limit: pass through. Over it: reject immediately with HTTP 429, cheaply, before any real work happens. Simple in spirit — surprisingly subtle once it has to be fast, distributed, and exact.
What the new pieces do
- Clientnormal
- A well-behaved caller making a steady, reasonable number of requests. Should never notice the limiter exists.
- Bursty Clientabuse
- A buggy retry loop or a scraper hammering you. The reason rate limiting exists — must be cut off cleanly.
Step 1 · The skeleton
A bouncer before the work
The expensive Upstream API shouldn’t waste a single cycle on a request that ought to be rejected. So the decision has to happen before it, as early as possible. Where?
A request that should be rejected must not waste the expensive upstream. Where does the limit check go?
By then the request has already consumed a connection and a thread — a flood still overwhelms the thing you’re protecting. The check must happen earlier.
You can’t trust clients — the whole point is to stop abusers and buggy retry loops that won’t cooperate. The limit must be enforced server-side, on a boundary you control.
Every request hits the limiter first: allow → forward, deny → 429 right there. The backend only ever sees approved traffic, so a flood never reaches the systems you’re protecting. Reject early, reject cheap.
Put the Rate Limiter at the API Gateway — the front door. Every request hits the limiter first. Allow → forward to the upstream service. Deny → return 429 Too Many Requests right there. The backend only ever sees approved traffic.
Why this piece earns its place
Checking at the front door quietly assumes there is only one door. That holds on the day you draw it, and stops holding the first time an internal service calls the Upstream API directly, a partner gets a private endpoint, or someone stands up a second ingress for webhooks. Every one of those paths is unlimited, and nothing in the diagram says so. So I would treat the gateway check as the coarse shed rather than the whole enforcement story: the upstream keeps a cheap internal quota of its own, and anything that can reach it without crossing the gateway is either routed back through the gateway or given its own limiter. The same assumption bites on identity. At the edge you know only what the request claims — a header, an API key, a source address — so a limit is exactly as strong as that claim, and on an unauthenticated path the best handle you have is an address an abuser can rotate. That is the real price of checking early: you get the cheapest possible rejection, and you pay for it in how little you know about who you are rejecting.
What the new pieces do
- API Gatewayedge
- Where every request enters. The natural, centralized place to decide who gets through and who gets a 429.
- Rate Limitercheck
- A fast check that runs before any real work: is this caller under their limit? If yes, pass; if no, reject with 429.
- Upstream APIservice
- The expensive backend you're protecting. It only ever sees requests the limiter has already approved.
Step 2 · The rules
Limit what, for whom?
"100 requests" is meaningless without two answers: per what key (user? IP? API key?) and how much (free vs paid, per-endpoint).
"100 requests" is meaningless alone. What two things must a limit specify?
Those don’t define who is limited or by how much. A limit needs an identity to count against and a number — not when or where.
The limiter builds a key — user:42, ip:1.2.3.4, key:abc:/search — and looks up that key’s limit from a Rule Store (free 100/min, paid 10k/min, strict endpoints stricter). The key decides what you’re actually protecting.
Method and size can refine a limit, but they don’t answer "whose quota?" — without a caller key you can’t attribute or enforce anything. Identity first, then the number.
Add a Rule Store. The limiter identifies the caller and builds a key — user:42, ip:1.2.3.4, key:abc:/search — then looks up that key’s limit. Free tier 100/min, paid 10k/min, expensive endpoints stricter. Rules are cached in the limiter so the lookup isn’t itself a bottleneck.
Why this piece earns its place
The moment rules are cached in the limiter, the Rule Store stops being the truth on the hot path and becomes the truth eventually. That is the right trade, but it has a shape worth naming out loud. A tightening you ship in the middle of an incident does not take effect when you save it; it takes effect a refresh interval later, instance by instance, and for that whole window the same caller gets a different limit depending on which gateway they land on. So the refresh interval is a genuine knob: short enough that an emergency rule lands in seconds, long enough that a large fleet is not polling the store as a matter of routine. Push invalidation if you have it, poll if you don’t, but pick the number deliberately rather than inheriting a library default. The other thing to settle now is precedence, because one key usually matches several rules at once — a tier limit, a per-endpoint override, and a temporary block on one abusive token. Decide which wins and write it down, or two instances will resolve the same key differently and you’ll spend an afternoon proving it.
- per-keyuser · ip · token
- tieredfree vs paid
- cachedrules
What the new pieces do
- Rule Storeconfig
- What the limits are: 100/min for free, 10k/min for paid, per-endpoint overrides. Loaded and cached by the limiter.
Back of the envelope
- key = user:42 / ip:… / token:…
- what you’re actually protecting
- tiered: free 100/min, paid 10k/min
- plus per-endpoint overrides
- rules cached in the limiter
- the lookup isn’t the bottleneck
Step 3 · The algorithm
How do you count?
You need to track each key’s recent usage. A naive fixed window ("max 100 per minute") is easy but lets a caller fire 100 at 0:59 and 100 more at 1:00 — a 2× burst across the boundary.
A fixed "100 per minute" window lets a caller fire 100 at 0:59 and 100 at 1:00 — a 2× burst. What counts better?
Halving the limit punishes every legitimate caller to paper over a boundary artifact, and the 2× burst across the new boundary still happens. Fix the counting scheme, not the number.
Exact, but storing every request’s timestamp per key is memory-heavy at scale. It works — token bucket / sliding-window counter get nearly the same correctness far cheaper.
Each key’s bucket refills steadily; a request spends a token, an empty bucket means deny. It allows healthy bursts while enforcing a long-run average — O(1) memory per key, µs per check. Sliding-window counters also fix the boundary.
Keep the counters in Redis (in-memory, microsecond ops). The favourite algorithm is the token bucket: each key has a bucket that refills at a steady rate; every request spends one token, and an empty bucket means deny. It allows healthy bursts while enforcing a long-run average. Sliding-window counters smooth out the boundary problem.
Why this piece earns its place
Two numbers fall out of this, and they do completely different jobs. The refill rate is the limit — a hundred a minute is about 1.67 tokens a second, and that is what holds over an hour. The capacity is how much unused allowance a caller may spend at once, and it is the one people set carelessly. Capacity is really a statement about the upstream, not about fairness: it is the largest instantaneous burst you are willing to hand the backend from a single key. Set it to the full 100, as here, and a caller idle for a minute can land a hundred requests in one breath — and a thousand such callers can do it in the same breath, all of them under their limit and collectively a spike. Set it to a single token and you have a strict rate, which sounds tidy right up until a legitimate client with a handful of parallel workers gets a 429 while sitting well under its quota. I would size capacity from what the upstream can absorb, keep the refill honest to the number we publish, and quote that rate to customers, because it is the one they can actually plan against.
- µsper check
- burstsallowed
- O(1)memory / key
What the new pieces do
- Redisstate
- In-memory counters/buckets keyed by caller. Reads and atomic increments here are sub-millisecond.
Back of the envelope
- “100 per minute” ⇒ refill 100 ÷ 60 ≈ 1.67 tokens/sec
- the same limit as the fixed window, expressed as a rate instead of an edge
- capacity 100 ⇒ one 100-request burst, then 1.67/sec
- bursts stay allowed, but the 2× boundary spike above becomes impossible
- state = 1 count + 1 timestamp ≈ 16 B/key
- O(1) per key — nothing grows with request volume
- 16 B × 10M active keys ≈ 160 MB
- the whole limiter fits in a single Redis instance
- µs per check in Redis
- in-memory atomic ops
Step 4 · Make it distributed
Many gateways, one truth
You run dozens of gateway instances behind a load balancer. If each counts in its own memory, a caller gets N × limit by spreading requests across them — the limit is a fiction.
Dozens of gateway instances each count in their own memory. What goes wrong, and how do you fix it?
A caller spreading requests across N instances gets N × the limit, because no instance sees the others’ counts. The limit becomes a fiction. You need one shared count.
Every instance reads/writes the same counter, and the read-modify-write is atomic (INCR with expiry, or a Lua script) so two instances incrementing at once can’t both slip under the limit. Shared + atomic = actually correct.
All-to-all gossip is slow, lossy and racy — counts converge too late to enforce a hard limit in real time. A single shared atomic counter is simpler and correct.
Centralize the count in shared Redis so every instance reads and writes the same counter. Crucially, do the read-modify-write atomically — INCR with expiry, or a small Lua script — so two instances incrementing at once can’t both slip under the limit (a classic race condition).
Why this piece earns its place
Sharing the count moves the problem rather than removing it: that counter now lives on exactly one Redis node, and the caller generating the most traffic is by definition generating the hottest key. Ordinary traffic spreads fine, because keys hash across the cluster and each caller is one key. An attack does not spread — a flood from one key concentrates on one slot, and that slot’s node has to absorb an atomic write per request while still serving everybody else’s counters. If surviving that matters, the move is to split one caller’s bucket into a handful of sub-buckets that hash to different slots, each holding a share of the limit, with an instance picking one; you give up a little exactness at the boundary to stop a single key being the whole limiter’s throughput ceiling. The matching discipline is that the Lua script must be the only thing that ever writes those keys. One forgotten code path that reads, decides and then increments puts back exactly the race the script exists to remove, and it will surface only as a caller who is occasionally a few requests over — which nobody reports.
- sharedcounter
- atomicINCR / Lua
- nodouble-spend
Back of the envelope
- 1 shared counter, all instances
- else a caller gets N × limit
- INCR / Lua = atomic
- increment + compare in one round trip
- no check-then-incr race
- two requests can’t both pass at 99
Step 5 · Go global
One Redis, or one per region?
The API now runs in three regions for latency. A caller in Tokyo hits the Tokyo gateway; a caller in Frankfurt hits Frankfurt’s. If each region has its own Redis, a caller who can reach multiple regions (or gets load-balanced across them) can rack up limit×regions in total throughput. If there’s one global Redis, every check crosses an ocean. Which do you pick?
Gateways run in three regions. Where does the counter live?
Every request now pays cross-region round-trip latency on the hot path — exactly what regional gateways existed to avoid. Correctness at the cost of the latency you just fixed.
A caller who can reach two regions effectively gets 2× the limit, and nothing ever reconciles it. Fine for coarse per-region shedding, not for a hard global cap.
Each region enforces a local sub-limit instantly (fast, no cross-region hop), while a background process aggregates regional counts into a global view used to tighten local sub-limits over time. Trades perfect global exactness for the latency the whole system was built around.
Split the limit: each region’s Redis enforces a fast local sub-limit (say, limit/3 per region) with no cross-region hop, while an async process aggregates the regional counts and adjusts sub-limits if one region is idle and another is saturated. The global total is approximately right, not exactly right — the same recall-vs-latency trade every distributed counter makes at global scale.
Why this piece earns its place
Splitting the limit three ways assumes the traffic splits three ways, and real traffic almost never does. A customer whose users all sit in one time zone sends nearly everything to one region, so they run into a third of the limit they are paying for and start collecting 429s while their global usage sits comfortably under the cap. That is the failure this step’s own answer creates, and unlike an approximate global count it is loudly customer-visible. The aggregation loop is what repairs it, which makes its interval the number that matters: it has to notice an idle region and lend that headroom to a saturated one before the burst it was meant to absorb is over, and for a spike measured in seconds that is a hard ask. So I would rather over-allocate than starve — give each region a share above the even split, so the shares sum to a little more than the global limit, accept that a caller can briefly exceed it, and let the aggregator claw the shares back. Over-admitting a paying customer slightly is a much cheaper mistake than throttling them at a third of what they bought.
Step 6 · Fast & resilient
Don’t let the limiter fail you
Now every request makes a network hop to Redis — adding latency to the hot path — and if Redis hiccups, does your whole API go dark?
Every request now hops to Redis, and if Redis hiccups your API could go dark. What do you do?
Blocking on a slow/unreachable Redis adds its latency (or timeout) to every request — the limiter becomes the outage. You need a fast path and an explicit failure policy.
No limiter means an abuser can melt your backend the moment Redis blips — you’ve removed the protection exactly when load is rising. Keep it, but make it fast and decide its failure mode.
A small per-instance cache syncs with Redis periodically, trading a little precision for speed. And you choose up front: fail-open (allow if Redis is down — availability) or fail-closed (deny — protection). Public APIs usually fail open; abuse-sensitive paths fail closed.
Cut the hop with a small local token cache per instance that syncs with Redis periodically, trading a little precision for speed. And decide your failure mode up front: fail-open (allow if Redis is down — favour availability) or fail-closed (deny — favour protection). Most public APIs fail open; abuse-sensitive paths fail closed.
Why this piece earns its place
The local cache is itself a way to exceed the limit, and it is worth saying how much. Each instance is holding tokens it has already claimed from Redis but not yet spent, so the true ceiling is the shared limit plus everything sitting uncommitted across every instance — meaning the limit loosens as you add gateways, which is precisely when you add them. That makes the lease size the knob: claim a small batch per instance and you stay close to exact but you are back to talking to Redis constantly; claim a large batch and you are fast and meaningfully over. I would size it from the overshoot I can tolerate rather than from the round trips I want to save, and hand unspent tokens back on a clean shutdown. The second thing to pin down is that the failure you actually meet is slow, not down. A Redis that answers in a second is worse than one that refuses instantly, so the client timeout is the real policy: it has to be small, and expiring it has to trigger the fail-open or fail-closed decision immediately rather than after everyone has already waited.
What the new pieces do
- Throttle Metricsobserve
- Tracks allow/deny ratios and top offenders, so you can tune limits and spot attacks instead of flying blind.
Step 7 · See it working
Observe, signal, tune
A silent limiter is dangerous: you can’t tell a healthy deny from a misconfigured one strangling real users, and clients have no idea why they’re blocked.
A silent limiter hides misconfigurations and leaves blocked clients confused. What do you add?
Metrics let you tell a healthy deny from one strangling real users and spot attacks; standard headers turn a blunt 429 into a contract ("N left, retry in T") so good clients self-throttle — cutting load more than blocking does.
A log nobody aggregates can’t show allow/deny ratios or top offenders in real time, and tells the blocked client nothing. You need live metrics and client-facing headers, not a passive log.
A 429 is normal operation, often thousands per second — emailing each is absurd. Aggregate into metrics and dashboards, and communicate limits to clients via response headers.
Emit metrics — allow/deny rates, top offenders — to spot attacks and tune limits. And be a good citizen to clients: return X-RateLimit-Limit, X-RateLimit-Remaining, and a Retry-After header on a 429 so well-behaved callers back off gracefully instead of retrying into the wall.
Why this piece earns its place
Retry-After is a promise, and the naive version of it manufactures its own incident. Throttle a large set of callers in the same second and hand every one of them the same number, and they all come back in the same second — the flood you just shed returns intact and now synchronized, which is worse than the unsynchronized one you started with. So the value wants jitter, spread over a window rather than pointing at a single instant, and the metrics want to show the shape of the returning traffic, not only the deny rate. X-RateLimit-Remaining is a promise too, and a harder one to keep honestly once the count is cached locally or held per region: that number reflects whichever counter this instance can see, so a client that plans against it can be told it has requests left and still be refused by the shared truth a moment later. I would under-report remaining rather than over-report it. And keep the caller key out of the metric labels — one time series per key is unbounded cardinality that can cost more than the limiter, so aggregate, and publish top offenders as a separate sampled list.
You did it
You just designed a rate limiter.
Everything you assembled, in order
- Check at the gateway, before the work — reject early with 429.
- A rule store maps each key (user/IP/token) to its limit.
- Token bucket in Redis: smooth average plus controlled bursts.
- Shared, atomic counters make the limiter correct across instances.
- Local caching for speed; fail-open vs fail-closed for resilience.
- Metrics plus Retry-After / X-RateLimit headers to observe and cooperate.