The whole design, in writing
Learn system design by building a service discovery system like Consul, etcd, Eureka or Kubernetes DNS step by step. An interactive guide covering why hardcoded IPs break in a dynamic fleet, a service registry with self-registration, health checks and heartbeats, client-side vs server-side discovery, backing the registry with consensus for HA, caching and watches, and the unhappy paths (stale entries, AP-vs-CP, split brain).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is service discovery?
In a microservices fleet, "payments" isn't one server — it's a shifting set of instances that autoscale up and down, get redeployed, move hosts, and die. When "orders" needs to call "payments", which IP does it use? Hardcode one and it's wrong within minutes. The addresses are never stable.
Service discovery lets services find each other by logical name instead of hardcoded address. Instances register their location in a registry; consumers look up the current healthy instances of a service and call one. Health checks keep the directory honest, and the registry itself is made highly available.
What the new pieces do
- Consumerclient
- A service that must call another. It knows the logical name ("payments") but not which instances are alive or where.
Step 1 · The hardcoded trap
Static IPs break
The simplest approach: put the target service's IP in a config file. It works on day one. Why does it fall apart in a real, dynamic fleet?
Hardcoded target IPs in config. Why does it break in a microservices fleet?
In a dynamic fleet, instances are ephemeral: autoscaling adds/removes them, deploys replace them, crashes kill them, schedulers move them. A hardcoded IP points at something that's often gone, and it can't spread load across the live set. Addresses must be looked up, not fixed.
Editing isn't the problem; the churn is. Even a perfectly-edited IP is stale minutes later when that instance is replaced or rescheduled. You need dynamic lookup, not better editing.
Storage is trivial. The real issue is that the mapping from service name to live instances changes constantly, so any static snapshot is wrong almost immediately.
Instances are ephemeral: autoscaling, deploys, crashes and rescheduling add and remove them constantly, so any hardcoded address is stale within minutes and can't spread load across the live set. In a dynamic fleet, the mapping from service name → live instances changes continuously. You must look it up, not bake it in.
Why this piece earns its place
The tempting cheaper answer is to keep one fixed address per service and point it at a load balancer — a virtual IP, or a single hostname the consumer never has to change. It genuinely does remove the churn from the consumer’s config, which is why it survived so long in static estates, and it is worth being able to argue against properly rather than dismissing it. Look at what sits behind that address. The load balancer still has to know which instances are alive right now, and maintaining that membership list is precisely the problem we just said no human process can keep up with. You have not removed discovery, you have relocated it into a box someone else configures — and if the list behind it is still hand-edited, autoscaling and rolling deploys invalidate it exactly as fast as they invalidated the config file. Done properly, with a registry behind it, this becomes the server-side option four steps from now. The lesson is narrower than addresses are bad: a stable name is fine, but a stable name backed by a stale membership list is the original bug wearing a hostname.
Step 2 · A live directory
The service registry
If addresses must be looked up, something has to hold the current map of name → live instances. What is it, and how do entries get in?
Introduce a service registry: a live directory mapping each service name to its currently-registered instance addresses. Instances put themselves in it (next step), and consumers query it to get the current set. The registry is the single source of truth for "who's out there right now" — the heart of discovery.
Why this piece earns its place
This step quietly assumes a name maps to one flat set of interchangeable instances, and that assumption holds for about a week. Then you are running two versions side by side through a rolling deploy, or a canary that should take a thin slice of traffic, or instances spread across zones where a consumer would much rather call the ones beside it, or a staging fleet that must never be reachable from production. A directory that stores only name and address cannot express any of that, so teams express it in the only field they have — the name. You get payments-v2, payments-canary, payments-euw1, every consumer hardcodes which variant it wants, and changing where traffic goes is a config edit and a redeploy again. So make an entry richer than an address from the very first version: every registration carries tags — version, zone, environment, weight — and a lookup filters on them. The name stays the stable thing consumers depend on, and everything that actually varies lives inside the entry, where it can be changed without touching the caller.
What the new pieces do
- Registrygateway
- The live directory: instances register here; consumers query it for the current healthy addresses of a service.
Step 3 · Getting in the book
Self-registration
How do the registry's entries appear and stay current as instances start and stop? Who's responsible for registering an instance?
How do instances get into (and out of) the registry as they start/stop?
Manual registration can't keep up with autoscaling and deploys happening every few seconds. Registration must be automatic.
Consumers can't know what's running — that's the entire problem discovery solves. Instances themselves (or a platform agent) must report their existence.
On boot, an instance registers its address with the registry; on graceful shutdown it deregisters. In self-registration the instance does this directly; in third-party registration a platform (like Kubernetes) registers pods for you. Either way it's automatic and tied to the instance lifecycle.
Instances self-register: on startup an instance tells the registry "payments is available at 10.0.0.7:9001", and on graceful shutdown it deregisters. (In the third-party variant, the platform — e.g. Kubernetes — registers/deregisters instances for you as pods come and go.) Registration is automatic and tied to the instance lifecycle, so the directory tracks the fleet without humans.
Why this piece earns its place
Self-registration answers the question is the process up, and consumers need the answer to a different one: can this instance serve a request. A process that has just started has empty connection pools, cold caches and unwarmed code paths; if it registers the moment it boots, the registry starts handing its address out seconds before it can do anything useful, and every rolling deploy produces a burst of timeouts nobody can reproduce afterwards. So gate registration behind the same readiness check the instance would expose to a probe, and only register once it passes. The second trap is subtler: what address does the instance report? An instance knows its own hostname, which behind NAT, inside a container, or on a multi-homed host is frequently not something any consumer can dial — it registers a value that is perfectly true and completely unreachable, and the entry looks healthy because nothing about it is malformed. Either have the platform supply the routable address, which is one of the strongest real arguments for third-party registration, or make the first successful check from outside the instance the thing that promotes an entry to usable.
- on startregister
- on stopderegister
- automatictied to lifecycle
What the new pieces do
- Instance Aservice
- One running instance of the target service. It self-registers its address on startup and heartbeats while alive.
- Instance Bservice
- Another instance. Autoscaling and deploys add and remove these constantly — their addresses are never stable.
- Instance Cservice
- A third instance. The registry must always reflect exactly which of these are currently healthy.
Step 4 · Only the living
Health checks & heartbeats
Graceful deregistration is the happy path — but instances also crash without a chance to deregister. If the registry keeps handing out a dead instance's address, consumers fail their calls. How does the registry know an instance is actually alive?
An instance crashes without deregistering. How does the registry stop handing out its address?
Either the registry actively probes each instance's health endpoint, or instances send periodic heartbeats with a TTL. An instance that fails checks or stops heartbeating is marked unhealthy and removed, so consumers only ever get live addresses.
Relying on consumers to notice failures is slow and pushes many failed calls before removal. The registry must proactively track liveness via checks/heartbeats.
Crashes skip deregistration, so "registered" and "alive" diverge constantly. The registry must continuously verify liveness, not trust the last registration.
Track liveness continuously. Either the registry actively probes each instance's health endpoint, or instances send periodic heartbeats with a TTL — miss a few and you're presumed dead. An instance that fails checks (or stops heartbeating) is deregistered, so consumers only ever receive addresses of instances that are healthy right now. Crash cleanup is automatic.
Why this piece earns its place
The knob this introduces is how long an instance may stay silent before you presume it dead — the heartbeat interval multiplied by the misses you tolerate. Set it long and a crashed instance keeps collecting traffic for that whole window, which is the failure the step exists to prevent. Set it short and you start evicting the living: a stop-the-world garbage collection pause, a slow disk, one dropped packet window, and a perfectly healthy process is deregistered, vanishes from the fleet, re-registers, and flaps. The way to set it is to measure your own pause and latency distribution and sit above its tail, then accept the remaining window as the price of not flapping. The direction that hurts most is the correlated one. If the registry loses its network rather than the instances losing theirs, every heartbeat stops at once, and a literal-minded TTL deregisters the entire fleet — a checker problem turned into a total outage. So cap it: when an implausibly large share of instances go quiet in the same moment, stop expiring and keep serving the last known set, because the far likelier explanation is that the thing doing the counting is wrong.
- heartbeat / probeprove liveness
- TTLmiss → presumed dead
- deregistercrash cleanup
What the new pieces do
- Health Checksservice
- Probes/heartbeats that tell the registry which instances are alive; dead ones are deregistered so no one calls them.
Step 5 · Who does the lookup?
Client-side vs server-side discovery
A consumer needs one healthy instance to call. Should the consumer query the registry and pick an instance itself, or should a router in the middle do it? This choice shapes where the discovery logic lives.
Where should the "query registry + pick an instance" logic live?
Client-side discovery is one valid option (fast, one less hop), but it puts discovery + load-balancing logic in every service and every language. The alternative — a router/proxy — keeps clients dumb. Both are legitimate; it's a tradeoff, not "always client".
Server-side discovery (a router/LB queries the registry) is valid and keeps clients simple, but adds a network hop and a component to run. It's not the only answer — client-side avoids the hop. It's a tradeoff.
In client-side discovery the consumer asks the registry and load-balances across instances itself — fewer hops, but logic in every client. In server-side discovery a router/proxy (or platform LB) queries the registry and forwards — dumb clients, but an extra hop and component. Pick per ecosystem; service meshes push this into a sidecar.
Two patterns. Client-side discovery: the consumer queries the registry, gets the instance list, and load-balances itself — fewer hops, but discovery logic lives in every client/language. Server-side discovery: a router/proxy (or platform load balancer) queries the registry and forwards the call — clients stay dumb, at the cost of an extra hop and a component to run. Both are valid; modern service meshes push this logic into a per-instance sidecar.
Why this piece earns its place
The extra hop is the wrong thing to decide this on — one more hop inside a datacenter is rarely what hurts. The cost that actually decides it is the upgrade path. Client-side discovery ships the registry client, the balancing policy and the retry behaviour inside every application binary, so the day you change how instances are weighted, or fix a bug in that library, the rollout is a redeploy of every service that depends on it, coordinated across teams who get nothing out of your change. Introduce a second language and you either reimplement all of it or run two implementations that quietly disagree about what healthy means. Server-side puts that logic in one place you can upgrade on your own schedule, and the bill is honest: the router now sits on the data path of every call, needs its own capacity and its own availability story, and when it fails everything fails together — whereas client-side degrades one consumer at a time. The sidecar is attractive on exactly this axis: the logic leaves application code, but it is deployed per instance, so upgrading it is a fleet operation instead of a pull request in thirty repositories.
- client-sideconsumer picks, 1 fewer hop
- server-siderouter picks, dumb client
- sidecarmesh factors it out
What the new pieces do
- Router / LBservice
- The server-side option: a proxy that consults the registry and load-balances, so consumers stay dumb.
Step 6 · The registry can't lie or die
Consensus & availability
The registry is now critical infrastructure that everything depends on. If it goes down, no one can discover anyone; if two copies of it disagree during a partition, services get contradictory routing. How do you make it both available and consistent?
Back the registry with a consensus system — etcd, Consul, or ZooKeeper (Raft/ZAB) — so a majority quorum agrees on the registry state and it survives node failure without losing data or splitting. This is a deliberate CP choice (consistent, may reject writes during a partition). The alternative, Eureka, chooses AP: stay available and serve possibly-stale data during partitions, betting that slightly-stale routing beats no routing. Pick per your tolerance — and either way, let consumers cache so a registry blip doesn't stop calls.
Why this piece earns its place
Consensus is the right home for the registry’s state and the wrong place to put the fleet’s churn. Every write through Raft or ZAB is a majority round trip that has to be durably logged before it counts, so if each instance’s heartbeat is a write, the registry’s load grows with the size of the fleet multiplied by the heartbeat rate — and the traffic costing the most carries the least information, namely still alive, nothing changed. That is how a quorum store gets tipped over by an ordinary deploy rather than by an outage. Keep liveness cheap and separate: a heartbeat lands on the node the instance is connected to, and only a genuine membership change — a registration, a deregistration, an instance crossing its threshold — is committed through consensus. The other thing a quorum quietly assumes is that the majority is nearby. Stretch one quorum across regions and every write pays a wide-area round trip, while a regional split leaves the minority side unable to write at all. Run a quorum per region and federate between them, so a region that loses its neighbours can still discover itself.
- quorummajority agrees
- CPconsistent (etcd/Consul/ZK)
- APavailable, stale (Eureka)
What the new pieces do
- Consensus Storedata
- The registry's replicated, consistent backing store, so the directory survives node failure without losing or splitting.
Step 7 · Fast and fresh
Caching, watches & DNS
Consumers can't query the registry on every single call — that's a lookup per request and huge load on the registry. But cached instance lists go stale as instances change. How do you get both speed and freshness?
Consumers cache the instance list locally (fast lookups, resilience if the registry blips) and stay fresh via watches/subscriptions — the registry pushes changes so caches update within seconds instead of polling. Expose discovery through familiar interfaces: many systems offer DNS-based discovery (resolve payments.svc to current instances) alongside a rich API. Kubernetes bakes this in — a Service name resolves via cluster DNS to healthy pod endpoints, with kube-proxy/the mesh load-balancing.
Why this piece earns its place
A watch inverts the freshness problem in a way that is easy to miss. With polling, a broken registry shows up as an error on every attempt. With a watch, it shows up as silence — and silence is indistinguishable from nothing has changed. A consumer whose subscription died an hour ago has a cache that looks perfectly healthy and a picture of the fleet that is an hour old, and it will go on calmly routing to instances that were retired long ago. So the subscription needs a cursor: the revision the client has already seen, sent on reconnect, so the registry either replays what was missed or explicitly tells the client the gap is too old and it must re-list. Being told is the important half — a client that silently resumes from a revision the registry has already compacted has lost changes without ever knowing. Then make cache age a first-class signal rather than an internal detail: every consumer should know, and be able to report, how long it has been since its view was last confirmed. Serving from a stale cache through a registry outage is the behaviour we want; not knowing the cache is stale is not.
- cacheno lookup per call
- watchpush updates in seconds
- DNStransparent resolution
Step 8 · The sharp edges
Stale entries, herds & split brain
Discovery has classic failure modes: an instance dies but lingers in caches (stale entry), a registry restart triggers a re-registration stampede, and a network partition can split the registry's view.
Tolerate stale entries at the caller with retries, timeouts and circuit breakers — assume any address might be dead and fail over to another instance. Smooth heartbeat/registration storms with jitter and sensible TTLs so a registry restart doesn't get hammered. Handle partitions per your CP/AP choice (quorum rejects a minority side; AP serves stale). And keep discovery decoupled from the call path via caching so the registry is never a synchronous dependency of every request.
Why this piece earns its place
The catch in tolerate stale entries at the caller is that retries are not free, and you need them most at the moment they cost most. A dead address in a cached list is not a rare event during a deploy — it is a steady share of every consumer’s calls, and if each of those calls quietly attempts a couple of alternates, a partly unhealthy fleet has its load multiplied onto the instances that are still standing, which is how one bad instance becomes an outage. So give retries a budget: cap them as a fraction of a consumer’s outbound traffic rather than as a fixed count per call, so a broadly failing dependency makes retrying stop rather than accelerate. Scope the breaker to the instance, not the service — a ghost address should be ejected from that one consumer’s local list while its siblings keep taking traffic, whereas a service-level breaker punishes healthy instances for a dead neighbour. And only replay what is safe to replay: a timeout does not tell you the instance never received the request, and blindly retrying a non-idempotent call on a second instance is how a stale-address annoyance becomes a data problem.
You did it
You just designed service discovery.
Everything you assembled, in order
- Hardcoded IPs break because instances constantly autoscale, deploy, crash and move.
- A service registry maps logical names to current live instance addresses — the phone book.
- Instances self-register on start and deregister on stop (or a platform registers them).
- Health checks and heartbeats deregister crashed instances so only live addresses are handed out.
- Client-side vs server-side discovery decides where lookup + load-balancing lives (sidecars split the difference).
- Back the registry with consensus (CP: etcd/Consul/ZK) or accept AP staleness (Eureka) — plus client caching.
- Cache + watch for fast, fresh lookups; DNS-based discovery (as in Kubernetes) makes it transparent.