The whole design, in writing
Learn system design by building a distributed file system like GFS or HDFS step by step. An interactive guide covering the single file-server ceiling, splitting huge files into large chunks across chunkservers, a single master holding the namespace and chunk map, keeping the master out of the data path, replication with rack-aware placement, the primary/pipelined write path, making the master survivable, and the unhappy paths (small files, stragglers, re-replication).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a distributed file system?
A data pipeline needs to store files that are terabytes each and read them at huge aggregate throughput across a fleet of worker machines. One file server can't hold them, can't feed a thousand readers at once, and loses everything when its disk dies.
A distributed file system (GFS, HDFS) presents one namespace over a whole cluster of commodity machines. It's tuned for a specific workload — huge files, big sequential reads and appends — and gets its scale from a clever split: a single master owns the metadata, while the actual bytes stream directly between clients and hundreds of chunkservers.
What the new pieces do
- Clientclient
- A data-processing job (think MapReduce) that reads and appends huge files in big sequential streams — not tiny random edits.
Step 1 · The ceiling
One file server, one workload
Start with a single big file server (an NFS box). Workers mount it and read. It's simple and correct. Where does it fall over for a data-crunching cluster?
A single file server feeding a cluster of workers. What caps it?
Naming is irrelevant. The real caps are capacity (one box's disks), aggregate throughput (one box's NICs feeding a thousand readers), and durability (its disk dies).
One server can't hold petabytes, can't feed a thousand workers at full bandwidth, and loses data when it fails. A cluster filesystem spreads all three across many machines.
RAM doesn't add disk capacity, network bandwidth to many readers, or redundancy. The bottleneck is fundamentally "one machine," which more RAM can't fix.
One box caps three things at once: capacity (its disks), aggregate throughput (its network feeding many readers), and durability (its disk failing). A data cluster needs the file's bytes spread across many machines so a thousand workers can each read their slice at full speed, and no single failure loses data.
Why this piece earns its place
The call to spread bytes across a cluster only pays off because of an assumption this step makes quietly: the workload is huge files, read sequentially and appended to rather than overwritten. Spreading is not free — every read now needs a lookup before it can start, so the first byte arrives later than it would from a single mounted server. That cost vanishes inside a hundred-megabyte stream and is ruinous on a 4KB one. So the day this design meets an interactive workload — millions of small files, random reads, latency measured per operation — it is worse than the one box it replaced, and no tuning fixes it, because the ask-then-stream shape is the architecture rather than a setting. The honest move is to name that boundary here instead of defending it at the end: this is a throughput filesystem for a batch cluster, and a latency-sensitive workload deserves a different store rather than this one bent to fit.
Step 2 · Big blocks
Split files into large chunks
To spread a terabyte file across machines, you break it into pieces. How big should a piece be — kilobytes like a normal filesystem block, or something far larger?
How big should each piece of a file be, and why?
Millions of tiny blocks explode the metadata the master must track and add per-block overhead. For huge sequential files, large chunks drastically cut metadata and maximize streaming throughput.
A whole-file piece can't be spread across machines or read in parallel, defeating the point. You want pieces big enough to be efficient but small enough to distribute.
Big chunks mean far fewer chunks per file, so the master's metadata is small enough to hold in memory, and each read is a long sequential stream — ideal for the workload. The trade is that small files waste a chunk, which the system accepts.
Split files into large chunks (GFS used 64 MB; HDFS blocks are often 128 MB). Big chunks mean a petabyte of data is only millions of chunks — few enough for the master to track in memory — and every read is a long sequential stream that saturates disk and network. The cost (a small file wastes a whole chunk) is a deliberate trade for this workload.
Why this piece earns its place
Large chunks concentrate load as surely as they shrink metadata, and that is the failure this answer creates. A small but popular file — a binary or a config every worker opens when a job starts — lands in a single chunk, so its handful of replicas absorb the whole cluster’s read burst in the same instant while thousands of other chunkservers sit idle. The fix is not a smaller chunk size; that would surrender the metadata win everywhere to solve a problem in a few files. It is to give that file more replicas than the default so the burst spreads across more machines, and to stagger job startup so a thousand workers don’t all ask in the same second. A client that is stuck waiting can also be pointed at another client that already pulled the bytes. The general lesson: chunk size is a placement decision as much as a storage one, because nothing smaller than a chunk can be spread at all.
Step 3 · A fleet to hold them
Chunkservers
Those chunks need to live somewhere that scales and survives failure. What holds them, and how do you make chunk loss survivable from the start?
Where do chunks live so the store scales and survives failures?
One copy per chunk means any server failure loses data, and high-end boxes waste money. Use commodity chunkservers and replicate each chunk across several of them.
Chunkservers store chunks as ordinary files and stream them to clients; replicating each chunk (default 3×) across different servers/racks makes any single failure survivable and spreads read load.
The master holds metadata only — the actual bytes are far too large for it and must live on the chunkserver fleet. Keeping data off the master is central to the design.
Run a fleet of commodity chunkservers that store chunks as ordinary local files and stream them to clients. Replicate each chunk across several chunkservers (default 3×) so any single failure is survivable and popular chunks can be read from multiple sources in parallel. Capacity and read bandwidth both scale by adding chunkservers.
Why this piece earns its place
Three is a knob, not a law, and you should be able to say how you would set it. One copy loses data on the first disk failure. Two survives a failure but leaves a single copy standing for however long the repair takes, so one more unlucky failure in that window — or a routine reboot of the machine holding it — loses the chunk. Three keeps a spare while the cluster heals. Going higher buys durability you cannot measure while costing a full extra copy of every byte stored and every byte written, and the write cost is the one people forget. It should also vary by data rather than being one global number: a scratch intermediate file a job can simply recompute doesn’t deserve the same protection as the input nobody can regenerate. Set it per file, and treat the default as the answer for data you would have to apologise for losing.
What the new pieces do
- Chunkserver 1data
- A commodity box storing large chunks as plain Linux files. Streams bytes directly to and from clients.
- Chunkserver 2data
- Holds replicas of chunks. Writes are pipelined replica-to-replica so bandwidth isn't wasted fanning out from one node.
- Chunkserver 3data
- The third replica (default replication is 3×). Placed on a different rack so one rack failure can't lose a chunk.
Step 4 · Keep the master out of the bytes
One master for metadata
Something must track which chunks make up each file and where each chunk's replicas live. But if every byte of every read flowed through that one component, it would instantly be the bottleneck. How do you get a single source of truth without it choking on data?
A single master holds the metadata. How do you stop it becoming the data bottleneck?
No single machine can serve a cluster's full read bandwidth — that's the ceiling you're escaping. The master must be removed from the data path, not scaled up.
Multiple masters streaming data reintroduces coordination and still funnels bytes through metadata nodes. The elegant answer keeps ONE master and routes data around it entirely.
The master answers "which chunkservers hold chunk N?" — a tiny control message — then the client talks to those chunkservers directly for the actual data. The master handles metadata only, so a single master serves a huge cluster. Clients cache locations to avoid re-asking.
Split control from data. The client asks the master a tiny question — "which chunkservers hold chunk N of this file?" — and the master answers from its in-memory map. Then the client streams the actual bytes directly to/from those chunkservers; the master never touches file data. One master can steer an enormous cluster because it only moves metadata.
Why this piece earns its place
Keeping the master out of the data path only holds if clients stop asking it, so they cache the locations they’re given — and a cache means a client’s view of the cluster goes stale exactly while the cluster is repairing itself. A replica moves, a chunkserver dies, and the client dials a machine that is gone or no longer holds that chunk. That has to fail fast and loudly rather than quietly return old bytes: the client retries another replica from its cached set, and only on a miss goes back to the master for a fresh answer. Two things keep the master’s side of this small. Cached locations carry an expiry, so a stale entry can’t live forever. And a client asks about several chunks in one request, so a job streaming a whole file pays one lookup, not one per chunk. The load the master really carries is proportional to file opens, not to bytes moved — that’s the number to quote when someone doubts one master is enough.
- 1 mastermetadata only
- in-RAMchunk map
- directclient ↔ chunkserver data
What the new pieces do
- Mastergateway
- The single metadata authority: holds the file→chunk namespace and which chunkservers hold each chunk. Answers "where?", then gets out of the way.
Back of the envelope
- control: client → master
- tiny location lookup
- data: client ↔ chunkservers
- bytes never hit the master
- clients cache locations
- master load stays low
Step 5 · Don't lose a chunk
Replication & rack-aware placement
Chunkservers fail daily, and sometimes a whole rack or switch goes down at once. Three replicas are useless if all three sit in the same rack. How do you place copies so real, correlated failures can't take out a chunk?
You keep 3 replicas of each chunk. Where do you place them?
Three replicas in one rack all die when that rack's switch or power fails — correlated failure. Placement must spread replicas across failure domains (racks) to be meaningful.
The master places replicas on different racks so a rack outage loses at most one copy, balances by disk usage, and re-replicates when a copy is lost. Rack-aware placement is what turns 3× into real durability.
Concentrating replicas on one server destroys the redundancy — that server's failure loses every copy. Replicas must be on distinct servers, ideally distinct racks.
The master does rack-aware placement: it puts a chunk's replicas on chunkservers in different racks so a rack/switch/power failure loses at most one copy. It balances placement by disk utilization, and when a replica is lost it schedules re-replication to restore the target count. Replication is only as good as the independence of where the copies sit.
Why this piece earns its place
Placing copies in different racks pays for durability with cross-rack bandwidth, and the bill arrives at the worst possible moment. Every write now crosses rack uplinks to reach its other replicas, and when a rack does fail, every chunk it held is under-replicated at once, so the master schedules a wave of copying across those same uplinks while surviving jobs are still trying to read. Healing at full speed can hurt more than the failure did. So repair is throttled: a cap on how many clone operations run at once, cluster-wide and per chunkserver, plus a bandwidth limit on each, so restoring copies never starves live traffic. The queue is ordered by risk rather than by arrival — a chunk down to one copy is cloned before a chunk down to two, and a chunk nobody reads can wait. Placement policy and repair policy are really one policy; designing either alone is how a rack failure becomes a cluster outage.
- 3×default replication
- rack-awareindependent copies
- autore-replicate on loss
Step 6 · Writing at scale
The primary & pipelined replication
A write must land on all three replicas consistently, and you don't want to waste the writer's bandwidth sending the same bytes three times, nor let three replicas disagree on the order of concurrent appends.
The master grants one replica a short lease as the primary; it picks the order of mutations so all replicas apply them identically. Data is pushed once and pipelined replica-to-replica (client → replica A → B → C) to use every link's full bandwidth instead of fanning out from the client. Because the workload is append-heavy, GFS offers an efficient record append that atomically adds to the end at a system-chosen offset.
Why this piece earns its place
A lease is a timeout, and a timeout can’t tell a dead primary from a slow one. A primary stalled in garbage collection, or sitting behind a partition, still believes it holds the lease while the master — watching that lease expire — is free to grant another. Now two replicas think they order writes. The lease alone does not close that gap; what closes it is a version number the master bumps on the chunk every time it grants a lease. A replica that missed mutations carries an older version, so the master won’t hand it to readers, won’t make it primary, and collects it as garbage. The stale writer’s traffic is refused because its version is wrong, not because anyone managed to detect that it was slow. Lease extensions then ride on heartbeats the master already exchanges, so the common case costs no extra round trip — and declining to extend is how the master stops mutations on a chunk it is about to move.
- leaseone primary orders writes
- pipelineA → B → C
- appendatomic record append
Step 7 · Protect the one brain
Make the master survivable
You deliberately built a single master — elegant, but now it's the one thing whose loss stops all new lookups. How do you keep a single logical master without it being a fatal single point of failure?
Persist every metadata change to a replicated operation log and take periodic checkpoints, so the master's entire state can be recovered by replay. Run shadow masters that follow the log and serve read-only lookups, and promote a standby quickly on failure. Keep chunk locations out of the persistent state — the master rebuilds them from chunkserver heartbeats at startup, so the log stays small. Because the master is out of the data path, a brief failover doesn't stop in-flight transfers.
Why this piece earns its place
A promoted master is not ready the moment its log finishes replaying. It has the namespace, but it deliberately never persisted where any chunk lives, so for its first stretch of life it is only listening to chunkservers report what they hold. If it starts making placement decisions during that silence, it concludes the cluster has lost nearly everything and launches a re-replication storm that copies the cluster onto itself. So a new master stays read-only until enough of the fleet has checked in, and only then may it schedule repairs — the same reason HDFS holds a starting NameNode in safe mode. The knob here is checkpoint frequency, because replay time is simply the time since the last checkpoint: checkpoint rarely and failover stretches out into log replay, checkpoint constantly and you stall live mutations unless the checkpoint is built in the background from a copy of the state. Recovery time is a number you choose, not one you discover mid-outage.
What the new pieces do
- Shadow Masterservice
- A hot standby replaying the master's operation log. Serves read-only lookups and can be promoted fast if the master dies.
- Op Log + Checkpointcontrol
- The master's durable, replicated operation log plus periodic checkpoints — the source of truth it recovers from.
Step 8 · The sharp edges
Small files, stragglers & healing
Real clusters hit the design's soft spots: millions of tiny files bloat the master's memory, a slow "straggler" chunkserver drags a job, and failures constantly erode replication.
Discourage small files (they waste chunks and inflate metadata) — pack them or use a layer on top. Combat stragglers by reading a chunk from the fastest of its replicas and by speculative task execution upstream. Let the master continuously re-replicate under-replicated chunks and rebalance disk usage as servers join and leave. Verify chunks with checksums to catch corruption, and rely on the append/at-least-once model rather than promising fully random-write consistency.
Why this piece earns its place
The append path is where this design hands work back to the application, and you should say so plainly. An atomic record append guarantees a record lands at least once on every replica, not exactly once: when one replica fails a mutation the client retries, and the replicas that succeeded on the first attempt keep their earlier copy. Replicas can also carry padding where a retry skipped past a region. So reading a file back is not simply reading bytes — records have to be self-describing. Each carries a checksum so a reader can walk past padding and garbage to the next real record, and a unique id so duplicates can be dropped. Readers filter; they never assume. The bargain is deliberate: making appends exactly-once would need a coordination round on every write, which is precisely what the lease exists to avoid. The system stays simple and fast by publishing a weaker guarantee and letting one shared client library absorb it.
What the new pieces do
- Re-replicationrepair
- Master-scheduled work that rebuilds under-replicated chunks onto healthy chunkservers after a failure.
You did it
You just designed a distributed file system.
Everything you assembled, in order
- One file server caps capacity, throughput and durability — spread files across a cluster.
- Split huge files into large chunks (64–128MB) so metadata stays tiny and reads stream.
- Store chunks on commodity chunkservers, each chunk replicated (3×) and disposable.
- A single master serves only metadata; clients stream bytes directly to/from chunkservers.
- Rack-aware placement makes 3 replicas survive correlated (rack) failures.
- A leased primary orders writes; data is pipelined replica-to-replica; appends are atomic.
- Make the master recoverable (op log + checkpoint + shadows), and continuously re-replicate.