Vibe Engines
YouTube
LEARNING PATH · Systems & Backend

Distributed Systems Deep Dive

For engineers who want consensus, replication and consistency in their bones.

Advanced ~6h 45m18 lessons20 steps

The hard core of distributed systems, threaded from the CAP theorem through Raft, quorums, vector clocks and CRDTs to real designs. Most protocols come with a simulator to drive them, and Raft, vector clocks, CRDTs and the hash ring add a challenge to implement one step yourself.

  • Reason precisely about consensus, replication and consistency
  • Implement Raft election, vector clocks and CRDT merges
  • Tune quorums and predict the consistency/availability they buy
  • Recognize the failure modes before they page you
0 / 18 done · 0%
  1. HandbookNext up

    CAP Theorem & Consistency Models

    The impossibility result that frames everything.

  2. Handbook

    CP vs AP Databases

    What CAP means when you actually pick a database.

  3. Handbook

    Partitioning, Sharding & Replication

    Split data, replicate it, keep it consistent.

  4. Handbook

    Consensus, Transactions & Coordination

    Getting nodes to agree — the core problem.

  5. Handbook

    Raft vs Paxos

    The two consensus protocols, compared.

  6. Lab

    Raft: The Election

    Drive a Raft cluster through elections and failures.

  7. Challenge

    Raft Leader Election (Majority)

    Implement one step of the Raft election.

  8. Lab

    The Quorum Dial

    Tune N/R/W and watch consistency vs availability move.

  9. ToolTool · optional

    Quorum (N/R/W) Explorer

    Compute quorum overlap for any N/R/W.

  10. Handbook

    Concurrency, Locks & Isolation Levels

    Locks, races, and the cost of coordination.

  11. Challenge

    Vector Clock Merge

    Order events without a global clock.

  12. Lab

    Vector Clocks: Who Caused What

    See causality captured by vector clocks.

  13. Challenge

    CRDT: Grow-Only Counter

    Merge conflicting replicas that always converge.

  14. Lab

    CRDT Merge: No Conflict

    Watch two divergent replicas reconcile, no coordinator.

  15. Lab

    The Rumor Mill: Gossip

    Spread state epidemically — anti-entropy gossip.

  16. Challenge

    Consistent Hashing Ring

    Implement the ring that shards and rebalances.

  17. ToolTool · optional

    Consistent Hashing Visualizer

    Visualize rebalancing when a node joins or leaves.

  18. System Design

    Design a Distributed Lock

    A lock that survives node death.

  19. System Design

    Design an ID Generator

    Unique IDs without coordination.

  20. System Design

    Design a Key-Value Store

    Assemble it all into a replicated KV store.

How this path works

This path is for engineers who already know the headline terms, CAP, quorums, Raft, eventual consistency, and want to understand them well enough to predict behavior: senior backend and infrastructure engineers, people working on databases or coordination layers, and anyone who has watched a split-brain or a stale read in production and wanted to know exactly why it happened.

Deep distributed-systems questions, whether in senior interviews or in real design reviews, rarely ask for definitions. They ask what a specific protocol does in a specific failure: a leader that pauses mid-term, a write acknowledged by two of three replicas, two replicas incrementing the same counter while disconnected, a lock whose lease expires while its holder is still working. Answering takes the mechanism, not the slogan.

The path builds that mechanism in layers: the impossibility results and the CP versus AP choice, then agreement through Raft, then quorums, then causality and conflict-free merging with vector clocks and CRDTs, then membership and placement with gossip and the hash ring. Raft, vector clocks, CRDTs and the hash ring each come with a challenge where you implement one step yourself, and most layers have a simulator you can drive through failures. Three designs at the end, a lock, an ID generator and a replicated key-value store, put the pieces back together.

What each step gives you

  1. CAP Theorem & Consistency Models

    The precise statement behind the slogans: what consistency and availability mean in CAP, why partition tolerance is not optional, why "CA" is a myth, the normal-operation trade-off PACELC adds, and the consistency spectrum from linearizable through causal to eventual. Read it carefully, because every later protocol in this path buys one point on that spectrum, and the page's brute-force quorum proof sets up the quorum steps.

  2. CP vs AP Databases

    What CAP means when you actually pick a database: CP systems refuse some requests during a partition, while AP systems keep serving possibly stale or conflicting data, and real platforms choose per data type, as the worked cart-versus-checkout scenario shows. It prepares you to justify a store choice in a design review and to explain why the trade-off only bites during an actual partition.

  3. Partitioning, Sharding & Replication

    Placement and copying, the two halves of every replicated store: partitioning schemes, consistent hashing with virtual nodes, hot partitions, and the single-leader, multi-leader and leaderless replication topologies that quorums tie together. In this path it bridges CAP and consensus, so read the replication table closely and note where one leader keeps things simple and where leaderless replication pushes conflict handling onto the application.

  4. Consensus, Transactions & Coordination

    The core problem stated properly: why agreement is hard to guarantee when messages can be delayed and nodes can crash, how majority quorums and leader election make progress in practice, what coordination services provide, two-phase commit versus sagas, and idempotency. It prepares you to say when a problem truly needs consensus and when a weaker tool, such as an idempotency key, will do.

  5. Raft vs Paxos

    The two consensus protocols compared on one design commitment: Raft's strong leader, which splits consensus into leader election, log replication and safety, against Paxos's value-at-a-time formulation with no built-in persistent leader. It covers their lineage, why etcd chose Raft, and which to reach for, so you can answer "why not Paxos?" with a reason rather than a reputation.

  6. Raft: The Election

    Run a five-node Raft cluster: followers wait on randomized election timers, the first to time out becomes a candidate, bumps the term and requests votes, and a majority makes it leader, which then sends heartbeats. Kill the leader and watch a fresh election, then use what you saw to explain why randomized timeouts make split votes rare and why each term has at most one leader.

  7. Raft Leader Election (Majority)

    Implement election_result(n, grants): count the candidate's own vote plus the granted ones and return whether that reaches a strict majority of the whole cluster, including even-sized clusters where half is not enough. The discussion after you solve it covers what a real node adds, such as one vote per term, the log up-to-date check and stepping down on a higher term, and how skipping any of them produces two leaders.

  8. The Quorum Dial

    Dial N, R and W, write a value to W replicas, read from R, and watch whether the two sets overlap. The pigeonhole argument becomes visible: when R + W exceeds N the read must touch a replica holding the latest write, and when it does not, you can produce a stale read on demand and see exactly why it happened.

  9. Quorum (N/R/W) Explorer

    The calculator companion to the simulator: for any N, R and W it reports whether reads are strongly consistent, how many failures reads and writes survive, and where the setting falls on the latency spectrum. Use it to work through the classic settings, majority, fast reads and fast writes, and to predict before you look what each one buys and gives up.

  10. Concurrency, Locks & Isolation Levels

    Coordination inside a single database before coordination across many: the races behind lost updates, pessimistic and optimistic locking, isolation levels and their anomalies, MVCC, and distributed locks with fencing tokens. It sits here because the fencing-token idea returns in the distributed lock design, and because seeing what serializable isolation costs on one machine explains why coordination across machines is expensive.

  11. Vector Clock Merge

    Implement the receive rule of vector clocks, receive(local, incoming, pid): take the element-wise maximum of your clock and the message's clock, then increment your own entry. Getting this one function right is what preserves causality, and it prepares you to explain how a system detects that two writes were concurrent instead of silently picking one of them.

  12. Vector Clocks: Who Caused What

    Trace three processes with no shared clock as they record local events, send messages carrying their vectors, and merge on receipt. Then compare vectors to decide which events happened before which and which were truly concurrent, the same comparison a replicated store makes when deciding whether two versions conflict or one simply supersedes the other.

  13. CRDT: Grow-Only Counter

    Compute the converged value of a grow-only counter, gcounter_value(payloads): merge every replica's per-replica vector by element-wise maximum, then sum the result. The small exercise carries the big idea, since a merge that is commutative, associative and idempotent lets replicas sync in any order, any number of times, and still agree without coordination.

  14. CRDT Merge: No Conflict

    Increment three replicas of a counter independently, merge them in whatever order you choose, and watch every replica land on the same value. It turns strong eventual consistency into something you have observed, and it prepares you to explain where CRDTs fit, such as counters and offline collaborative edits, and why they avoid conflicts rather than resolving them after the fact.

  15. The Rumor Mill: Gossip

    Start a rumor at one node and let every informed node tell a random peer each round: the informed set roughly doubles, so the whole cluster knows in about log N rounds, with no coordinator and no single point of failure. It prepares you to explain how stores like the key-value design at the end of this path track membership, and why a lost message barely matters.

  16. Consistent Hashing Ring

    Build a hash ring with virtual nodes using the provided hash32, then route each key to the first ring point at or after its hash, wrapping past the end. Implementing it settles details that hand-waving skips, such as the wrap-around, why production rings binary-search the sorted points instead of scanning them, and why too few virtual nodes leave the load uneven.

  17. Consistent Hashing Visualizer

    After implementing the ring, watch one rebalance: add and remove nodes, change virtual-node counts, and see which keys move and how load per node shifts. Use it to verify your mental model of the challenge you just solved, especially the claim that adding a node moves only about K/N keys and that more virtual nodes smooth out hot spots.

  18. Design a Distributed Lock

    Why a "locked = true" row fails, then the repairs in order: atomic acquire, TTL leases, ownership tokens, fencing tokens that make a lock actually safe, consensus-backed locks for high availability, renewal and granularity, and the clock and pause problems that remain. It prepares you to argue when a lock needs consensus, and when you should avoid a distributed lock altogether.

  19. Design an ID Generator

    Unique IDs without coordinating on every ID: a Snowflake-style 64-bit layout with timestamp, machine ID and sequence, safe one-time assignment of node bits, and what to do when the clock moves backward. In this path it shows the opposite strategy to consensus, coordinating once at startup instead of on every operation, and it prepares you to explain why the IDs are only roughly ordered.

  20. Design a Key-Value Store

    The capstone that assembles the path: consistent hashing for placement, replication along a preference list, quorum reads and writes, conflict resolution, gossip membership, hinted handoff and read repair. Read it as a review of every earlier step, and test yourself on the hard questions it ends with, such as how a read can return stale data even when R + W exceeds N.

A study plan

  1. Week 1

    Read CAP Theorem and Consistency Models, CP vs AP Databases and Partitioning, Sharding and Replication. Then write one paragraph for each datastore you use at work: what it does during a partition, and which consistency model it actually offers. Check your claims against its documentation.

  2. Week 2

    Consensus week: read Consensus, Transactions and Coordination and Raft vs Paxos, drive the Raft simulator through at least three leader failures, and solve the Raft leader election challenge. Finish by explaining on paper why two leaders can never exist in the same term.

  3. Week 3

    Quorums and causality: the quorum simulator and explorer, the concurrency handbook, then the vector clock challenge and lab and the CRDT challenge and lab. Produce a stale read on purpose in the quorum simulator and write down exactly which setting allowed it.

  4. Week 4

    Membership, placement and assembly: the gossip simulator, the hash ring challenge and visualizer, then the distributed lock, ID generator and key-value store designs. For each design, list which earlier steps it reuses, and answer its interviewer questions before reading the answers.

Questions about this path

How long does it take to learn distributed systems?

The core ideas in this path, consistency models, replication, quorums, consensus, causality and CRDTs, take most engineers four to eight weeks of steady study to understand well enough to reason with. Real fluency takes longer and comes from operating systems that fail in interesting ways, reading postmortems, and implementing small pieces yourself. That is why this path pairs reading with simulators and challenges: running a Raft election or merging vector clocks by hand builds intuition faster than rereading definitions.

Should I learn Raft or Paxos first?

Start with Raft. It was designed for understandability, splitting consensus into leader election, log replication and safety, and many newer systems that need consensus, such as etcd, use it. Once Raft makes sense, Paxos is easier to read as the older, more general formulation of the same majority-quorum idea. For most engineering work you need to know what these protocols guarantee and what they cost, such as needing a majority of nodes up to make progress, more than you need every detail of either one.

What is the difference between vector clocks and CRDTs?

Vector clocks track causality: they let a system tell whether one event happened before another or whether two events were concurrent, which is how a store detects conflicting versions. CRDTs are data types designed so that concurrent updates never conflict at all, because their merge is commutative, associative and idempotent. Both use an element-wise maximum over per-replica vectors, which is why they are often taught together, but one detects conflicts while the other avoids them. This path has a challenge and a simulator for each.

Do I need distributed systems knowledge for system design interviews?

For most system design rounds, a working knowledge of replication, partitioning, quorums and CAP is enough. Senior rounds, and infrastructure or database roles, often go deeper: expect questions about leader election, split-brain, lease-based locks, fencing tokens and conflict resolution under concurrent writes. This path is aimed at that deeper level. If your interview is soon and general, the Pass the System Design Interview path is a faster route, and this one makes a good follow-up.

Consensus, replication, and consistency are the hard core of distributed systems, and hand-waving them shows. This path takes you deep — from the CAP trade-offs through quorums, replication, and consensus protocols — until you can reason about failure and agreement in your bones, not just recite the definitions.

← All learning paths