The big idea

What is a distributed file system?

A data pipeline needs to store files that are terabytes each and read them at huge aggregate throughput across a fleet of worker machines. One file server can't hold them, can't feed a thousand readers at once, and loses everything when its disk dies.

A distributed file system (GFS, HDFS) presents one namespace over a whole cluster of commodity machines. It's tuned for a specific workload — huge files, big sequential reads and appends — and gets its scale from a clever split: a single master owns the metadata, while the actual bytes stream directly between clients and hundreds of chunkservers.

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the diagram grow, and at the end lose a chunkserver and kill the master to see why the master never touches your bytes. Hit Begin.

Step 1 · The ceiling

One file server, one workload

Start with a single big file server (an NFS box). Workers mount it and read. It's simple and correct. Where does it fall over for a data-crunching cluster?

Design decision: A single file server feeding a cluster of workers. What caps it?

The call: Capacity, aggregate read throughput and durability all cap at one box. — One server can't hold petabytes, can't feed a thousand workers at full bandwidth, and loses data when it fails. A cluster filesystem spreads all three across many machines.

One box caps three things at once: capacity (its disks), aggregate throughput (its network feeding many readers), and durability (its disk failing). A data cluster needs the file's bytes spread across many machines so a thousand workers can each read their slice at full speed, and no single failure loses data.

Workload-specific by design: GFS/HDFS aren't general-purpose. They assume huge files, sequential reads, and appends over random writes — and optimize hard for that. Designing to a known workload is what lets them be simple and fast where a general filesystem couldn't.

Step 2 · Big blocks

Split files into large chunks

To spread a terabyte file across machines, you break it into pieces. How big should a piece be — kilobytes like a normal filesystem block, or something far larger?

Design decision: How big should each piece of a file be, and why?

The call: Large chunks (e.g. 64–128 MB) so metadata stays tiny and reads stream. — Big chunks mean far fewer chunks per file, so the master's metadata is small enough to hold in memory, and each read is a long sequential stream — ideal for the workload. The trade is that small files waste a chunk, which the system accepts.

Split files into large chunks (GFS used 64 MB; HDFS blocks are often 128 MB). Big chunks mean a petabyte of data is only millions of chunks — few enough for the master to track in memory — and every read is a long sequential stream that saturates disk and network. The cost (a small file wastes a whole chunk) is a deliberate trade for this workload.

Why chunks are huge: Chunk size is a metadata-vs-waste dial. Larger chunks → less metadata and longer streams (great for big files), but more wasted space for small ones. Since the target workload is huge files, you crank chunk size way up and accept the small-file cost.

Step 3 · A fleet to hold them

Chunkservers

Those chunks need to live somewhere that scales and survives failure. What holds them, and how do you make chunk loss survivable from the start?

Design decision: Where do chunks live so the store scales and survives failures?

The call: On many commodity chunkservers, each chunk replicated across several. — Chunkservers store chunks as ordinary files and stream them to clients; replicating each chunk (default 3×) across different servers/racks makes any single failure survivable and spreads read load.

Run a fleet of commodity chunkservers that store chunks as ordinary local files and stream them to clients. Replicate each chunk across several chunkservers (default 3×) so any single failure is survivable and popular chunks can be read from multiple sources in parallel. Capacity and read bandwidth both scale by adding chunkservers.

Commodity, replicated, disposable: The design assumes constant hardware failure and makes chunkservers cheap and disposable. Durability comes from replication across the fleet, not from expensive reliable hardware — the same philosophy as object storage.

Step 4 · Keep the master out of the bytes

One master for metadata

Something must track which chunks make up each file and where each chunk's replicas live. But if every byte of every read flowed through that one component, it would instantly be the bottleneck. How do you get a single source of truth without it choking on data?

Design decision: A single master holds the metadata. How do you stop it becoming the data bottleneck?

The call: Client asks master only for chunk locations, then streams bytes directly to/from chunkservers. — The master answers "which chunkservers hold chunk N?" — a tiny control message — then the client talks to those chunkservers directly for the actual data. The master handles metadata only, so a single master serves a huge cluster. Clients cache locations to avoid re-asking.

Split control from data. The client asks the master a tiny question — "which chunkservers hold chunk N of this file?" — and the master answers from its in-memory map. Then the client streams the actual bytes directly to/from those chunkservers; the master never touches file data. One master can steer an enormous cluster because it only moves metadata.

Master out of the data path: This is the key idea. Because the master only serves small location lookups (and clients cache them), a single master isn't a throughput bottleneck. The data plane (client↔chunkserver) scales independently of the control plane (client↔master). One brain, many muscles.

Step 5 · Don't lose a chunk

Replication & rack-aware placement

Chunkservers fail daily, and sometimes a whole rack or switch goes down at once. Three replicas are useless if all three sit in the same rack. How do you place copies so real, correlated failures can't take out a chunk?

Design decision: You keep 3 replicas of each chunk. Where do you place them?

The call: Across different racks/failure domains, tracked by the master. — The master places replicas on different racks so a rack outage loses at most one copy, balances by disk usage, and re-replicates when a copy is lost. Rack-aware placement is what turns 3× into real durability.

The master does rack-aware placement: it puts a chunk's replicas on chunkservers in different racks so a rack/switch/power failure loses at most one copy. It balances placement by disk utilization, and when a replica is lost it schedules re-replication to restore the target count. Replication is only as good as the independence of where the copies sit.

Failure domains: Durability isn't just "how many copies" but "how independent are they." Spreading replicas across racks (and availability zones) ensures failures are uncorrelated, so 3 copies really do survive the failures you'll actually see. The master owns this placement policy.

Step 6 · Writing at scale

The primary & pipelined replication

A write must land on all three replicas consistently, and you don't want to waste the writer's bandwidth sending the same bytes three times, nor let three replicas disagree on the order of concurrent appends.

The master grants one replica a short lease as the primary; it picks the order of mutations so all replicas apply them identically. Data is pushed once and pipelined replica-to-replica (client → replica A → B → C) to use every link's full bandwidth instead of fanning out from the client. Because the workload is append-heavy, GFS offers an efficient record append that atomically adds to the end at a system-chosen offset.

Lease + pipeline: The lease gives a single primary temporary authority to serialize writes (consistency) without asking the master per-write. Pipelining forwards bytes down a chain so replication doesn't bottleneck on one node's uplink. Control decisions are cheap and centralized; data movement is distributed.

Step 7 · Protect the one brain

Make the master survivable

You deliberately built a single master — elegant, but now it's the one thing whose loss stops all new lookups. How do you keep a single logical master without it being a fatal single point of failure?

Persist every metadata change to a replicated operation log and take periodic checkpoints, so the master's entire state can be recovered by replay. Run shadow masters that follow the log and serve read-only lookups, and promote a standby quickly on failure. Keep chunk locations out of the persistent state — the master rebuilds them from chunkserver heartbeats at startup, so the log stays small. Because the master is out of the data path, a brief failover doesn't stop in-flight transfers.

Recoverable, not immortal: You don't make the single master never fail — you make its state durable and fast to recover. Op log + checkpoint = the whole namespace can be rebuilt; heartbeats re-derive volatile location data. HDFS later added an active/standby NameNode with a shared edit log to automate this.

Step 8 · The sharp edges

Small files, stragglers & healing

Real clusters hit the design's soft spots: millions of tiny files bloat the master's memory, a slow "straggler" chunkserver drags a job, and failures constantly erode replication.

Discourage small files (they waste chunks and inflate metadata) — pack them or use a layer on top. Combat stragglers by reading a chunk from the fastest of its replicas and by speculative task execution upstream. Let the master continuously re-replicate under-replicated chunks and rebalance disk usage as servers join and leave. Verify chunks with checksums to catch corruption, and rely on the append/at-least-once model rather than promising fully random-write consistency.

Design for the unhappy path: Tiny files → the master's memory is the scarce resource, so avoid them. Slow node → read another replica. Constant failures → continuous re-replication and rebalancing. The system is engineered to heal and self-balance faster than a big cluster breaks.

You did it

You just designed a distributed file system.

• O — n
• S — p
• S — t
• A —
• R — a
• A —
• M — a