The whole design, in writing
Learn system design by building an API gateway like Kong, Amazon API Gateway or Apigee step by step. An interactive guide covering the direct-to-microservice mess, one front door that routes to services, offloading cross-cutting concerns (auth, rate limiting, TLS) to the edge, request aggregation / backend-for-frontend, response caching, keeping the gateway stateless and redundant, and the unhappy paths.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is an API gateway?
You broke a monolith into a dozen microservices — great for the backend. But now the mobile app has to know a dozen hostnames, implement auth against each, handle a dozen retry policies, and re-deploy every time a service moves. Every client re-implements the same plumbing, and every service re-implements auth, rate limiting and logging.
An API gateway is a single front door in front of all your services. Clients call one endpoint; the gateway authenticates, rate-limits, routes each request to the right service, and can even compose several services into one response. Cross-cutting concerns live in one place instead of scattered across every client and every service.
What the new pieces do
- Clientsclient
- Browsers, mobile apps and third-party integrations. They want one simple, stable API — not a map of your internal services.
Step 1 · The mess
Clients talking to every service
Without a gateway, each client calls each microservice directly. The mobile app hardcodes a dozen endpoints, re-implements token handling for each, and breaks whenever a service is split or moved. Every service, meanwhile, re-writes auth, rate limiting and logging. What's the underlying problem?
Clients call a dozen microservices directly. What is the core problem?
Speed isn't the issue here — coupling is. Clients are bound to your internal service topology, and every service duplicates the same cross-cutting plumbing.
Clients must know every service and re-implement auth/retries; each service re-implements auth/rate-limit/logging. Both problems vanish if one component sits in front and owns the edge concerns.
More services makes the coupling and duplication worse, not better. The fix is a single front door, not a different number of services.
Two coupled problems: clients are welded to your internal topology (they must know every service and where it lives), and every service re-implements the same cross-cutting concerns (auth, rate limiting, TLS, logging). Both dissolve if one component owns the edge.
Why this piece earns its place
Both problems are real, but they bill you on different schedules. The coupling bill arrives the day you split a service. Server code you redeploy in an afternoon; a shipped mobile app you cannot. An install base keeps calling last year’s topology for as long as people decline the update, so “clients know every service” quietly means “no internal address may ever change again”. The duplication bill arrives on the day of a security fix. Twelve token checks are twelve code paths, owned by different teams on different release trains, and the fix is only finished when the slowest of them ships — until then the weakest implementation is your actual security posture. That is the tell for a cross-cutting concern: if a correct change to it needs N coordinated deploys, it is in the wrong place. The test worth applying before moving anything to the edge is whether all twelve services genuinely want the same answer; the ones that don’t belong to the service, not the door.
Step 2 · One front door
A single entry point
You want clients to call one stable endpoint, regardless of how many services exist behind it or how they're split. Where does that consolidation happen?
How do you give clients one stable API over many changing services?
A client library still couples clients to your topology and must be re-shipped to every app on every change — impossible for third parties. The indirection needs to live on the server side.
That throws away the independent scaling and deploys you split them out for. You want one API surface without one codebase.
The gateway is the single public endpoint. Clients know only the gateway; it holds the routing table and forwards each request to the right internal service, which can move or split freely.
Introduce the gateway as the single public endpoint. Clients know only api.example.com. The gateway holds the routing rules and forwards each call to the right service — which can be moved, split, renamed or rescaled without any client ever noticing.
Why this piece earns its place
It is worth being honest about what a single public endpoint commits you to. That hostname becomes the most permanent name you own: third parties hardcode it, partners put it in change-controlled config, and some of them will never edit that line again. So keep the host generic and push everything variable behind it — the version into the path, the client type into a header — because anything encoded in the name is frozen there with it. The other thing this bakes in is that one contract suits every caller. It does, until you need a partner API with a different auth model and different quotas, or an internal admin surface that must never be reachable from the public internet. At that point you either overload the one door with path conventions and hope nobody guesses the internal prefix, or you run a second gateway with its own rules. Run the second gateway. A boundary whose meaning depends on which path you knocked on is not a boundary.
What the new pieces do
- API Gatewaygateway
- The single entry point for every API call. Authenticates, rate-limits, routes, and shields the mesh of services behind it.
Step 3 · Routing
Which service handles this call?
A request for GET /orders/42 arrives at the gateway. It has to reach the orders service — and only the orders service — even as instances come and go. How does the gateway decide?
A call to /orders/42 arrives. How does the gateway send it to the right service?
Broadcasting wastes every other service's capacity and is ambiguous when more than one could respond. Routing should be a deterministic map from request to service.
The gateway maps /orders/* → orders service, /users/* → users service, and so on. Rules can key off path, host, method or headers, and forward to a healthy instance of the target.
Making the client name the service re-couples it to internal topology — the exact thing the gateway removes. The gateway owns routing so clients don't have to.
The gateway routes by matching each request against rules — usually the path (/orders/* → orders service), sometimes host, method or headers — and forwards to a healthy instance of the target service. Add, split or rescale services by editing routes, not clients.
Why this piece earns its place
The routing table is ordered, overlapping config, and it is the most dangerous file you own. /orders/* and /orders/*/refunds can belong to different services; first match wins, so a general rule sitting above a specific one makes the specific one dead without erroring. The traffic simply arrives somewhere that answers 404 — and that 404 comes from a service which never owned the path, so the symptom points away from the cause. Match by longest prefix rather than file order where the gateway allows it, review the table like code, and roll a change to a slice of instances before the pool. The quieter failure is rewriting. Strip /orders on the way in and the service builds its redirects, Location headers and pagination links in its own terms, which reach the client as paths that do not exist at the edge. If you rewrite the request you have also taken on rewriting the response, or you publish your internal path shape by accident.
What the new pieces do
- Users Serviceservice
- One microservice among many. Owns its own data and does one thing well, addressed by a path prefix.
- Orders Serviceservice
- Another service. The gateway routes /orders here and enforces its own timeout so it can't stall everything else.
- Catalog Serviceservice
- Product catalog. Cacheable, mostly-read — a good candidate for response caching at the gateway.
Step 4 · One place to check identity
Authentication at the edge
Every service needs to know who is calling and whether they're allowed. Re-implementing token validation in all twelve services means twelve chances to get security wrong. Where should auth happen?
Every service needs to authenticate callers. Where should that logic live?
Auth in every service duplicates security-critical code twelve times — twelve chances for a subtle bug, and inconsistent enforcement. Centralize the check.
A client can't be trusted to authorize itself; the server must verify. Auth is a server-side gate, and the gateway is the natural place for it.
The gateway authenticates every request at the edge — validating a JWT (locally, via a cached public key) or an opaque token against the identity service — then forwards the verified user identity to the service, which can trust it.
Do authentication at the gateway. It validates the caller's token on every request — a JWT checked locally against the identity service's public key (no round trip), or an opaque token introspected against the identity service — then passes the verified identity to the downstream service, which trusts the edge. One consistent gate instead of twelve.
Why this piece earns its place
Notice what this hands downstream: the services now trust a header. That trust is worth exactly as much as the network boundary underneath it. The moment anything can reach a service without passing the gateway — a debug port, a second ingress, a neighbouring pod, a batch job someone wired straight to the service — that header is forgeable by whoever is inside, and the caller gets to name themselves. So the edge check only works paired with something that makes the edge unavoidable: network policy or mTLS so a service accepts connections from the gateway alone, or an identity the gateway signs and the service verifies, or simply forwarding the original token so the service can check the signature itself. Choose one deliberately and write it down, because this failure does not look like a breach in any log. It looks like an ordinary request with an ordinary user on it, and the twelve services you spared from writing auth code are precisely the twelve with no code that would notice.
- 1 gatenot 12
- JWTverified at edge
- identitypassed downstream
What the new pieces do
- Identityauth
- Issues tokens on login and holds the keys. The gateway verifies tokens against it (or its public key) on every request.
Step 5 · Protect the services
Rate limiting & quotas
A buggy client in a retry loop, or an abusive third party, can hammer your API and take down the very services behind it. You want to cut them off before they reach a service. Where and how?
An abusive caller is hammering your API. Where do you stop them?
By the time the request reaches a service, it's already consumed connections and work. And every service would re-implement limiting. Reject at the edge, before the mesh.
The gateway tracks each client's request rate in Redis (a token bucket) and returns 429 once they exceed their quota — rejecting abuse before it touches any service, consistently across all routes.
A hard permanent ban is too blunt — it punishes shared IPs and legitimate bursts. Rate limiting throttles gracefully (429 + Retry-After) and is per-client, not a permanent block.
Rate-limit at the gateway. Keep per-client counters — a token bucket in Redis so the limit holds across all gateway instances — and return 429 Too Many Requests (with Retry-After) once a caller exceeds its quota. Abuse is shed at the edge, before it ever reaches a service, uniformly across every route.
Why this piece earns its place
A per-client quota is a fairness device, and it is easy to mistake it for a capacity device. Add up every client’s allowance and the total is usually several times what the services behind you can serve, so an ordinary day where many well-behaved callers are busy at once looks, from a service’s side, exactly like abuse — and every one of those requests is inside its quota, so the limiter waves them all through. Fairness needs a second ceiling beside it: an in-flight or concurrency cap per route that the gateway holds regardless of who is asking. The other decision to make now, rather than during the incident, is what the gateway does when the counter store is slow or unreachable. Fail closed and a Redis blip becomes a total outage of an API that was otherwise perfectly healthy; fail open and the limit disappears exactly when something is already going wrong. Fail open on a store error, and alarm loudly — the limiter exists to protect the services, not to become the thing that takes them down.
- 429over quota
- token bucketper client
- edgereject before mesh
What the new pieces do
- Rate Limitercache
- Per-client counters (token bucket) in Redis. Lets the gateway reject abusive callers before they touch a service.
Step 6 · Fewer round trips
Aggregation & caching
A mobile home screen needs data from three services. Making the app fire three separate calls over a slow mobile network is wasteful, and hammering the catalog service for the same rarely-changing data on every request is pointless. Can the edge help?
Two edge optimizations. Aggregation (a Backend-for-Frontend): the gateway fans out to several services and composes one response, so a mobile screen makes a single call instead of three. Response caching: cacheable GETs (like catalog reads) are served from a cache at the gateway, so repeat calls never touch the service. Fewer round trips for the client, less load for the services.
Why this piece earns its place
Aggregation changes the shape of the risk. One client call becomes three concurrent ones, so the composed response is as slow as the slowest branch and fails if any branch fails — you have multiplied the ways a single screen can break while making it look to the client like one call. So design the composition to degrade: give each branch its own deadline, return the sections that answered, and mark the one that didn’t, so the app renders a home screen with a missing shelf instead of an error page. That means the response shape has to permit absence, which is a decision taken at design time and impossible to retrofit mid-incident. It also changes what a client retry costs: the app sees one failed call and retries it, which is three more service calls at the moment things are already struggling. The caching half needs a narrower rule than it looks: only responses identical for every caller belong in an edge cache, and anything that varied with who asked either goes into the key or stays out entirely.
- 1 callvs 3
- cachehot GETs
- BFFshape per client
What the new pieces do
- Response Cachecache
- Caches cacheable GET responses (like catalog reads) so repeat calls never touch the service at all.
Step 7 · Don't become the bottleneck
Keep the gateway stateless & pooled
Every request now flows through the gateway. If it holds state or runs as one box, you've built a single point of failure and a scaling ceiling in front of everything. How do you keep it from being the weak link?
Keep the gateway stateless — counters and sessions live in Redis, config is pushed in — so you can run many identical instances behind a load balancer and scale them horizontally. Any instance handles any request; losing one costs a retry. The gateway becomes a pool, not a chokepoint.
Why this piece earns its place
“Stateless” is a claim about user state, and the gateway still holds plenty of the other kind: the routing table, the verification keys, the quota config. Those arrive by push, which means a pool is only eventually identical — during a rollout some instances route the old way and some the new, and any request can land on either. Make that survivable rather than surprising: version the config, change routes additively so old and new tables both answer correctly through the overlap, and push to a slice before the fleet. The sharp edge is that fast propagation cuts both ways. The property that lets you fix a route everywhere in seconds is the same property that breaks every route everywhere in seconds. So an instance handed config it cannot parse must keep serving the last good copy and report itself unhealthy — never start empty. An empty routing table is a gateway that is up, passing its own health check, and returning 404 for the entire API.
Step 8 · The sharp edges
Discovery, timeouts & versioning
Real systems shift under you: service instances come and go, a slow service threatens to stall the gateway, and your public API has to change without breaking existing clients.
Use a service registry so the gateway always routes to healthy, current instances (service discovery). Enforce per-route timeouts + circuit breakers so one slow service fails fast and in isolation instead of tying up gateway connections. And version the public API (/v1, /v2) so you can evolve it while old clients keep working. Watch the gateway from becoming a monolith of business logic — keep it thin.
Why this piece earns its place
Of the three, versioning is the one whose cost lands somewhere other than where you paid it. /v1 is a single routing rule at the edge, but it is a permanent behaviour contract inside the services, which now carry two code paths through every future change. Publishing a version hires maintenance forever unless you decide up front how it ends: announce the sunset when you announce the version, meter traffic per version per client so you can name who is still on the old one instead of guessing, and prefer additive changes so most releases need no new version at all. The tempting shortcut is to hide the whole problem at the edge — translate /v1 requests into the /v2 shape so the services only ever see one contract. It genuinely is cheaper, and it is also how a thin gateway starts growing a brain. If you do it, confine the shim to renaming and reshaping fields, never to deciding anything, and give it the same expiry date as the version it serves.
What the new pieces do
- Service Registrydiscovery
- The live map of which service instances are healthy and where. The gateway asks it where to send each route.
You did it
You just designed an API gateway.
Everything you assembled, in order
- Direct client-to-service calls couple clients to topology and duplicate cross-cutting concerns.
- One front door gives clients a stable API while services evolve freely behind it.
- Path/host routing sends each request to the right service via the registry.
- Authentication at the edge — verify a JWT once, pass identity downstream — instead of 12 times.
- Rate limiting with a Redis token bucket sheds abuse before it reaches any service.
- Aggregation (BFF) and response caching cut round trips and service load.
- Keep the gateway stateless + pooled; add discovery, timeouts, circuit breakers and versioning.