The big idea
What does a load balancer do?
A single server has a hard ceiling — a fixed number of requests per second before latency spikes and it falls over. Buy a bigger box and you've only raised the ceiling; you still have one machine that, when it dies, takes the whole service with it.
A load balancer lets you run many servers behind one address. It spreads incoming requests across the pool so no single box is overwhelmed, and quietly routes around any server that fails. Clients see one endpoint; behind it, capacity and redundancy scale by adding boxes.
How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the diagram grow, and at the end kill a server and kill the balancer to see what survives. Hit Begin.
Step 1 · The ceiling
One server, one ceiling
Everything runs on a single app server. As traffic climbs it saturates CPU and connections, latency balloons, and if it crashes the site is simply down. What's the real fix — a bigger server, or a different shape?
Design decision: One server is saturating under load and is a single point of failure. Best fix?
The call: Run many servers and spread traffic across them (scale out). — Horizontal scaling removes both problems at once: capacity grows by adding boxes, and one failure just means fewer boxes, not an outage — provided something spreads the traffic.
Scale out, not up: run many servers and share the load. That needs one component in front to decide, per request, which server handles it — and to notice when a server has died. That component is the load balancer.
Scale up vs scale out: Up = a bigger machine (simple, but a hard ceiling and a single point of failure). Out = more machines (capacity grows linearly and failures are survivable). Load balancing is what makes "out" possible.
Step 2 · One door
A single entry point
Clients can't be told "connect to server 4 of 12" — the set of servers changes constantly as you scale and as boxes fail. They need one stable address. Who stands at that door?
Design decision: The server pool changes constantly. What do clients connect to?
The call: A load balancer at a single address that forwards to a backend. — Clients hit one endpoint; the balancer holds the live list of servers and forwards each request to a healthy one. The pool can change freely behind it.
Put a load balancer at one address. Clients connect to it; it forwards each request to a chosen backend from a pool it manages. The pool can grow, shrink and lose members without a single client ever changing what it connects to.
Indirection: The balancer is a level of indirection: one stable public endpoint mapped to a fluid set of private servers. Every hard part — which server, is it alive, how to spread load — moves behind that one door.
Step 3 · Who gets it?
Which server should this request go to?
The balancer has a pool of servers and a request in hand. Send them all to server 1 and you've balanced nothing. Pick badly and one server melts while others idle. What's a good rule for choosing?
Design decision: A request arrives and the pool has 3 servers. How does the balancer choose one?
The call: A balancing algorithm — round-robin, least-connections, weighted, or hashing. — Round-robin rotates evenly; least-connections favors the least-busy server (great when request cost varies); weighted respects unequal server sizes; hashing pins a key (like client IP) to a consistent server. Pick per workload.
Use a balancing algorithm. Round-robin rotates through servers evenly. Least-connections sends the next request to the least-busy server (best when request durations vary). Weighted variants give bigger boxes more traffic. Hashing (e.g. by client IP or URL) consistently maps a key to the same server.
Pick the algorithm to the workload: Uniform, cheap requests → round-robin. Long-lived or variable-cost requests → least-connections. Heterogeneous servers → weighted. Need the same client/key on the same server (cache locality, affinity) → consistent hashing.
Step 4 · Interchangeable boxes
Any server, any request
Least-connections and failover both assume any server can handle any request. But if server 3 holds your logged-in session in its local memory, routing your next request to server 5 logs you out. What has to be true for servers to be interchangeable?
Design decision: For the balancer to route any request to any server, what must be true?
The call: Servers are stateless — session/state lives in a shared store. — If servers hold no per-user state, any of them can serve any request. Move session and data into a shared store (Redis/DB) so the app tier is a pool of identical, disposable boxes.
Make servers stateless: no per-user state on the box. Push session and application state into a shared store (a database or Redis). Now every server is identical and disposable — the balancer can route any request anywhere, add capacity by cloning, and lose a box without losing anyone's session.
Statelessness is the enabler: Statelessness is what makes horizontal scaling, load balancing and failover all work. If the box holds nothing precious, killing it costs nothing but a retry. State belongs in a store built to be shared and durable.
Step 5 · Don't route to the dead
Health checks
A server can crash, hang, or get wedged while still accepting connections. If the balancer keeps sending it requests, users hit errors and timeouts. How does the balancer know a server is actually able to serve?
Design decision: A server crashes but the balancer doesn't know. How does it stop routing there?
The call: Actively probe each server on an interval; evict it after N failed checks. — A health check hits a lightweight endpoint (or opens a TCP connection) every second or two; after a few consecutive failures the server is pulled from the pool, and re-added once it passes again. Detection is automatic and bounded in time.
Run continuous health checks. The balancer probes each server on an interval — a TCP connect, or an HTTP GET /health. After N consecutive failures it evicts the server from the pool; when it passes again it's added back (ideally with slow-start so it isn't instantly overwhelmed). Requests only ever go to servers proven healthy right now.
Active vs passive checks: Active: the balancer proactively probes a health endpoint. Passive: it watches real traffic and ejects a server that starts erroring/timing out. Good balancers do both — passive reacts instantly to real failures, active confirms recovery before restoring traffic.
Step 6 · Don't become the bottleneck
Who balances the balancer?
You removed the single point of failure from the app tier — and quietly created a new one. The load balancer is now the single box every request flows through. If it dies, everything behind it is unreachable. How do you make the balancer itself redundant?
Design decision: The single load balancer is now the single point of failure. How do you fix it?
The call: Run a standby balancer that takes over a floating VIP on failure. — Two balancers share a virtual IP via a heartbeat; if the active one dies, the standby claims the VIP in seconds and traffic continues. Clients never change the address they connect to.
Run the balancer in a redundant pair sharing a floating virtual IP. A heartbeat keeps the standby watching the active one; if it stops responding, the standby claims the VIP within seconds and serving continues — clients keep hitting the same address. (At larger scale you go further: multiple balancers behind anycast or DNS, so there's no single active box at all.)
No single anything: The rule generalizes: every layer that all traffic flows through must itself be redundant. Active-standby with a floating VIP is the classic pattern; anycast and DNS-based distribution scale it out so the balancer tier is a pool too.
Step 7 · When you can't be stateless
Sticky sessions & L4 vs L7
Sometimes a workload genuinely wants the same client on the same server (an in-progress upload, a local cache). And sometimes the balancer needs to route by content — path, host, cookie — not just spread connections. How much should the balancer understand?
Support session affinity ("sticky sessions") when needed — pin a client to a server via a cookie or consistent hash — but prefer stateless + shared store; stickiness re-introduces a little of the fragility you removed. And choose the layer: an L4 balancer routes by IP/port (fast, protocol-agnostic); an L7 balancer parses HTTP and can route by path/host/header, terminate TLS, and retry — more power, a little more cost.
L4 vs L7: L4 (transport): forwards TCP/UDP by IP:port, blazing fast, knows nothing about the request. L7 (application): understands HTTP, so it can do path/host routing, TLS termination, sticky cookies, header rewrites and smart retries. Many stacks use L4 at the edge and L7 deeper in.
Step 8 · The sharp edges
Failover storms & draining
Real failures cascade: a server dies and its load slams onto the survivors (possibly toppling them too — a failover storm); a recovered box gets flooded the instant it rejoins; a deploy kills servers mid-request.
Keep headroom so survivors can absorb a failed peer's share. Slow-start a recovered or new server so it ramps up instead of being crushed on rejoin. Use connection draining (graceful shutdown): stop sending new requests to a server being removed but let in-flight ones finish. Cap retries with timeouts and circuit breakers so one slow backend can't stall the whole balancer.
Design for the unhappy path: Server dies → survivors need spare capacity, not a domino. Server rejoins → slow-start, don't flood. Deploy → drain, don't drop. Slow backend → timeout + break the circuit. A balancer that only works when every backend is healthy is a demo.
You did it
You just designed a load balancer.
- Scale out, not up — many servers survive failure and grow capacity linearly.
- One stable entry point hides a fluid pool of backends from clients.
- Balancing algorithms — round-robin, least-connections, weighted, hashing — pick the server per workload.
- Stateless servers + a shared store make any box able to serve any request.
- Health checks (active + passive) evict dead servers and slow-start recovered ones.
- Make the balancer itself redundant — floating VIP + standby, or anycast/DNS at scale.
- Sticky sessions and L4-vs-L7 for the cases that need them; drain, headroom and circuit breakers for failures.