The whole design, in writing
Learn system design by building a load balancer like NGINX, HAProxy or AWS ELB step by step. An interactive guide covering the single-server bottleneck, a pool of stateless servers behind one entry point, balancing algorithms (round-robin, least-connections, weighted, hashing), health checks that evict dead servers, L4 vs L7 routing, making the balancer itself redundant, session affinity vs statelessness, and the unhappy paths.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What does a load balancer do?
A single server has a hard ceiling — a fixed number of requests per second before latency spikes and it falls over. Buy a bigger box and you've only raised the ceiling; you still have one machine that, when it dies, takes the whole service with it.
A load balancer lets you run many servers behind one address. It spreads incoming requests across the pool so no single box is overwhelmed, and quietly routes around any server that fails. Clients see one endpoint; behind it, capacity and redundancy scale by adding boxes.
What the new pieces do
- Clientsclient
- A flood of users hitting one hostname. None of them should know or care which server actually answers.
Step 1 · The ceiling
One server, one ceiling
Everything runs on a single app server. As traffic climbs it saturates CPU and connections, latency balloons, and if it crashes the site is simply down. What's the real fix — a bigger server, or a different shape?
One server is saturating under load and is a single point of failure. Best fix?
Vertical scaling raises the ceiling but it's still one machine with a hard limit — and still a single point of failure. When it dies, everything dies with it.
Optimization buys headroom but not infinite headroom, and it does nothing for the single-point-of-failure problem. One box is one box.
Horizontal scaling removes both problems at once: capacity grows by adding boxes, and one failure just means fewer boxes, not an outage — provided something spreads the traffic.
Scale out, not up: run many servers and share the load. That needs one component in front to decide, per request, which server handles it — and to notice when a server has died. That component is the load balancer.
Why this piece earns its place
Scaling out assumes the work actually divides, and here it does, because each request is independent of the others. What it does not divide is whatever those servers all sit in front of. If the saturated box was already blocked waiting on one database, one file system, one licence server, then ten copies of it block on the same thing and the graph barely moves — while that shared dependency now carries ten times the connections and may get worse. So before adding boxes I want to know what the old one was actually waiting on: CPU and connection limits are cured by more machines, contention behind them is not. That is also the honest limit of everything that follows. The pool is elastic; the tier behind it usually is not. Step 4 pushes session state into a shared store for good reasons, and that store then absorbs the traffic of the whole pool — it needs its own capacity plan, not an assumption that it scales because the thing in front of it does.
Step 2 · One door
A single entry point
Clients can't be told "connect to server 4 of 12" — the set of servers changes constantly as you scale and as boxes fail. They need one stable address. Who stands at that door?
The server pool changes constantly. What do clients connect to?
Client-side lists go stale the moment you add or lose a server, and push balancing logic into every client. You want one stable address that hides the pool.
Clients hit one endpoint; the balancer holds the live list of servers and forwards each request to a healthy one. The pool can change freely behind it.
DNS round-robin is crude balancing but can't health-check, reacts slowly (cached records keep sending traffic to dead boxes), and gives no per-request control. A real balancer does all three.
Put a load balancer at one address. Clients connect to it; it forwards each request to a chosen backend from a pool it manages. The pool can grow, shrink and lose members without a single client ever changing what it connects to.
Why this piece earns its place
One door only helps if what is behind it is accurate, and this step quietly assumes the balancer knows the pool. Somebody has to tell it. The cheap version is a static list in a config file, which is fine while servers are long-lived and someone edits that file on every change — and it fails in both directions you would expect: a new server sits idle because nobody added the line, and a decommissioned one keeps taking traffic until somebody remembers to delete it. Once instances come and go on their own, membership has to come from something that already knows — a service registry, an autoscaling group, the orchestrator’s endpoint list — and the balancer has to reload it without dropping live connections. Worth saying out loud: the health checks in step 5 are not a substitute for this. They decide whether a known server is serving right now; they cannot discover a server nobody told the balancer about.
What the new pieces do
- Load Balancerbalancer
- The single entry point. Picks a healthy backend for each request and forwards it — spreading load so no one server is overwhelmed.
Step 3 · Who gets it?
Which server should this request go to?
The balancer has a pool of servers and a request in hand. Send them all to server 1 and you've balanced nothing. Pick badly and one server melts while others idle. What's a good rule for choosing?
A request arrives and the pool has 3 servers. How does the balancer choose one?
That's not balancing — it hot-spots one box while the rest idle. You want to spread load, and ideally account for how busy each server already is.
Random is a real strategy and evens out over time, but it ignores how loaded each server actually is, so a slow request can pile onto an already-busy box. Least-connections adapts to real load.
Round-robin rotates evenly; least-connections favors the least-busy server (great when request cost varies); weighted respects unequal server sizes; hashing pins a key (like client IP) to a consistent server. Pick per workload.
Use a balancing algorithm. Round-robin rotates through servers evenly. Least-connections sends the next request to the least-busy server (best when request durations vary). Weighted variants give bigger boxes more traffic. Hashing (e.g. by client IP or URL) consistently maps a key to the same server.
Why this piece earns its place
Least-connections has one sharp edge worth naming, because it is the algorithm people reach for by default. It infers load from a count, and a server that is failing fast has the lowest count in the pool: it refuses or errors in a millisecond, its connections clear instantly, so it permanently looks like the idlest box and the balancer sends it more than anyone. The pool’s errors funnel into its sickest machine. The defence is not a different algorithm, it is pairing this one with the passive health checking of step 5, so a backend whose error rate spikes gets ejected instead of rewarded. The counter also has to measure the right thing. With keep-alive and HTTP/2 a single connection carries many requests, so connection count stops tracking work at all — count in-flight requests, or weight by observed latency, and treat the raw connection count as the proxy it is.
- round-robineven rotation
- least-connfollows real load
- hashkey → server
What the new pieces do
- Server 1service
- One interchangeable app instance. Holds no session state, so any request can go to any server.
- Server 2service
- Another identical instance. Add or remove copies freely to scale with traffic.
- Server 3service
- The pool grows horizontally: more boxes = more capacity and more failure headroom.
Back of the envelope
- round-robin / weighted
- spread by count or capacity
- least-connections
- adapts to variable request cost
- consistent hashing
- sticky key → server, cache-friendly
Step 4 · Interchangeable boxes
Any server, any request
Least-connections and failover both assume any server can handle any request. But if server 3 holds your logged-in session in its local memory, routing your next request to server 5 logs you out. What has to be true for servers to be interchangeable?
For the balancer to route any request to any server, what must be true?
Local session state pins a user to one server; if that server dies or the balancer routes elsewhere, the session is lost. State on the server breaks interchangeability.
If servers hold no per-user state, any of them can serve any request. Move session and data into a shared store (Redis/DB) so the app tier is a pool of identical, disposable boxes.
Broadcasting session to all servers is a consistency and bandwidth nightmare that gets worse as the pool grows. Centralize state instead of replicating it everywhere.
Make servers stateless: no per-user state on the box. Push session and application state into a shared store (a database or Redis). Now every server is identical and disposable — the balancer can route any request anywhere, add capacity by cloning, and lose a box without losing anyone's session.
Why this piece earns its place
Be precise about this in the room: stateless does not mean the state disappeared, it means it moved, and it moved into one shared thing that every server now depends on. You made the app tier disposable by making the store indispensable, so the store inherits the availability requirement the app tier just shed and has to be replicated for exactly the reason the servers are. There is a per-request cost too — what used to be a local memory read is now a network hop on the path of every request that touches a session. Two things follow. Keep what you put there small and hot: an identifier and a little context, not a serialized object graph. And notice that for pure identity you may not need the read at all — a signed token carried by the client proves who someone is with no lookup anywhere, at the price of not being able to revoke it the instant you want to. Server-side sessions cost a hop and buy instant revocation; tokens are the same trade run backwards.
Step 5 · Don't route to the dead
Health checks
A server can crash, hang, or get wedged while still accepting connections. If the balancer keeps sending it requests, users hit errors and timeouts. How does the balancer know a server is actually able to serve?
A server crashes but the balancer doesn't know. How does it stop routing there?
Manual removal is far too slow — thousands of requests hit the dead box first. Detection must be automatic and continuous.
A single failure is noisy (a transient blip isn't a dead server). You want an active, repeated probe with a threshold, not a reaction to one bad request.
A health check hits a lightweight endpoint (or opens a TCP connection) every second or two; after a few consecutive failures the server is pulled from the pool, and re-added once it passes again. Detection is automatic and bounded in time.
Run continuous health checks. The balancer probes each server on an interval — a TCP connect, or an HTTP GET /health. After N consecutive failures it evicts the server from the pool; when it passes again it's added back (ideally with slow-start so it isn't instantly overwhelmed). Requests only ever go to servers proven healthy right now.
Why this piece earns its place
Two knobs here, pulling against each other. The interval and the failure count set the detection window — a dead box keeps receiving traffic for roughly the interval times N, so at ~1–2s and a few consecutive failures that is seconds of errors — while tightening them toward zero means a garbage-collection pause or one slow probe flaps a healthy server out of the pool. Flapping is expensive: every eviction shifts that server’s share onto the survivors, which makes their probes slower, which evicts more of them. The second knob is what the probe asks. A /health that returns 200 straight out of the web framework proves the process is listening, not that it can serve — its connection pool to the store can be exhausted while the check stays green. A deeper check catches that but couples every server’s health to one shared dependency, so a store blip marks the whole pool dead at once. So put a floor under it: when every backend is failing the check, keep routing anyway. An empty pool is a guess that returns nothing to everybody.
- ~1–2sprobe interval
- N failsthen evict
- slow-startramp recovered box
What the new pieces do
- Health Checkshealth
- Continuously probes each server. A failing box is pulled from the pool so no request is routed to a dead server.
Step 6 · Don't become the bottleneck
Who balances the balancer?
You removed the single point of failure from the app tier — and quietly created a new one. The load balancer is now the single box every request flows through. If it dies, everything behind it is unreachable. How do you make the balancer itself redundant?
The single load balancer is now the single point of failure. How do you fix it?
"More reliable" still fails eventually, and when it does everything is down. You need a second balancer that can take over, not just a sturdier single one.
Two balancers share a virtual IP via a heartbeat; if the active one dies, the standby claims the VIP in seconds and traffic continues. Clients never change the address they connect to.
Client-side failover pushes complexity into every client and reacts slowly; many clients cache the dead address. Failover should be invisible, handled by the balancer pair and a shared VIP.
Run the balancer in a redundant pair sharing a floating virtual IP. A heartbeat keeps the standby watching the active one; if it stops responding, the standby claims the VIP within seconds and serving continues — clients keep hitting the same address. (At larger scale you go further: multiple balancers behind anycast or DNS, so there's no single active box at all.)
Why this piece earns its place
The failure this step introduces is the one the heartbeat cannot see. Losing the heartbeat means one of two things — the active balancer died, or the link between the pair died — and the standby cannot tell which, so the obvious rule, take the VIP when the heartbeat stops, can end with both balancers holding the same address and answering for it. That is split brain: connections land on whichever box the network learned most recently, and anything the pair keeps per-connection — sticky mappings, drain status, in-flight state — diverges between them. The fix is to not decide on a single signal: a second heartbeat over separate wiring, or a witness outside both boxes that arbitrates the claim, plus fencing so the loser releases the address before the winner takes it. The other thing I would raise is that a standby which only ever stands by is untested — its config drifts from the active one and you find out during the failure. Fail over deliberately, on a schedule, while someone is watching.
- VIPshared address
- heartbeatdetects failure
- ~secondsto fail over
What the new pieces do
- Floating VIProuting
- A single virtual IP that the active balancer holds. On failover the standby claims it, so clients never change what they connect to.
- Standby LBbalancer
- A second balancer in lockstep. If the active one dies, it takes over the VIP within seconds.
Step 7 · When you can't be stateless
Sticky sessions & L4 vs L7
Sometimes a workload genuinely wants the same client on the same server (an in-progress upload, a local cache). And sometimes the balancer needs to route by content — path, host, cookie — not just spread connections. How much should the balancer understand?
Support session affinity ("sticky sessions") when needed — pin a client to a server via a cookie or consistent hash — but prefer stateless + shared store; stickiness re-introduces a little of the fragility you removed. And choose the layer: an L4 balancer routes by IP/port (fast, protocol-agnostic); an L7 balancer parses HTTP and can route by path/host/header, terminate TLS, and retry — more power, a little more cost.
Why this piece earns its place
Stickiness makes a promise it can only keep while the pool is stable, and how you implement it decides how badly it breaks. Pin clients with a plain hash over the list of servers and the mapping is a function of that list: evict one box — which step 5 will — and the list is shorter, so nearly every client rehashes somewhere new and the entire pool loses its affinity at the precise moment one server has already gone. That is what consistent hashing is for; only the failed server’s share moves and everybody else stays put. Cookie-based affinity survives a pool change instead, but has its own gap: a client that does not return the cookie is simply unpinned, so there has to be a sane answer for that request rather than an error. And since stickiness exists for the cases you could not make stateless, design the fallback so losing affinity costs a re-fetch or a re-upload, never a session.
What the new pieces do
- Shared Storedata
- Session and app state live here, not on the servers — which is exactly what lets any server handle any request.
Step 8 · The sharp edges
Failover storms & draining
Real failures cascade: a server dies and its load slams onto the survivors (possibly toppling them too — a failover storm); a recovered box gets flooded the instant it rejoins; a deploy kills servers mid-request.
Keep headroom so survivors can absorb a failed peer's share. Slow-start a recovered or new server so it ramps up instead of being crushed on rejoin. Use connection draining (graceful shutdown): stop sending new requests to a server being removed but let in-flight ones finish. Cap retries with timeouts and circuit breakers so one slow backend can't stall the whole balancer.
Why this piece earns its place
Headroom is the one number this step actually asks you to pick, so here is how I would pick it. If a pool of n servers has to survive losing k of them, each survivor must carry its share while only n−k remain — which means the steady state runs at (n-k)/n of what the pool could take, and the difference is capacity you pay for every hour against an event that is rare. The arithmetic is why pool size changes the answer: surviving one loss out of a large pool costs a sliver of every box, while surviving one loss out of three costs a third of everything, permanently. Small pools are expensive to make resilient, which is a real argument for more and smaller servers rather than fewer big ones. Two cautions on measuring it. Size against the survivors’ knee, the load where latency turns upward, not average CPU, which looks calm right until it is not. And size for the correlated loss — a rack or an availability zone takes a slice of the pool at once, not one box.
You did it
You just designed a load balancer.
Everything you assembled, in order
- Scale out, not up — many servers survive failure and grow capacity linearly.
- One stable entry point hides a fluid pool of backends from clients.
- Balancing algorithms — round-robin, least-connections, weighted, hashing — pick the server per workload.
- Stateless servers + a shared store make any box able to serve any request.
- Health checks (active + passive) evict dead servers and slow-start recovered ones.
- Make the balancer itself redundant — floating VIP + standby, or anycast/DNS at scale.
- Sticky sessions and L4-vs-L7 for the cases that need them; drain, headroom and circuit breakers for failures.