The whole design, in writing
Learn system design by building an object store like Amazon S3 step by step. An interactive guide covering the single-disk baseline, a flat bucket/key namespace, splitting metadata from data, distributing objects across storage nodes, durability via replication vs erasure coding, read-after-write consistency, multipart uploads for huge objects, lifecycle tiering hot→cold, and the unhappy paths (hot keys, checksums, background repair).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is object storage?
You need to store an effectively unlimited number of files — photos, videos, backups, logs — of any size, reachable from anywhere, that you can never lose. A folder on a server's disk fails on all three: it fills up, it dies with the disk, and it's unreachable when the box is down.
Object storage (S3, GCS, Azure Blob) stores blobs by key behind a simple HTTP API: PUT to store, GET to read. Behind that flat interface it spreads and replicates every object across thousands of machines to deliver near-infinite capacity and famous "eleven nines" of durability (99.999999999%).
What the new pieces do
- Clientclient
- An app storing or fetching a blob — a photo, a backup, a video. Talks plain HTTP: PUT to store, GET to read, by a key.
Step 1 · The naive store
Files on one disk
Start with the obvious: save each upload as a file on one server's disk. It works for a demo. Name the three ways it fails as you grow.
Files on a single server's disk. What are the fundamental limits?
Naming is trivial to solve. The real limits are capacity (one disk fills), durability (a dead disk loses everything), and availability (a down box is unreachable) — all inherent to one machine.
Speed isn't the core issue. Even a fast single disk can't grow past its size, survive its own failure, or stay reachable when the server is down.
One disk fills up; when it dies your data dies with it; when the box is down nobody can read. Object storage exists to break all three ceilings by spreading data across many machines.
One disk means one ceiling on capacity, one point of data loss, and one point of unavailability. To store unlimited data you can't lose and can always reach, the bytes must live on many machines. Everything else follows from that.
Why this piece earns its place
The alternative everyone reaches for before distributing anything is a bigger box: more disks in one chassis, RAID across them, and a nightly backup. It is worth saying out loud why that loses, because at a glance it looks like it covers the same three ceilings. RAID survives a disk, not the machine — the controller, the power supply and the rack are all still single copies, and a rebuild on a full array runs for hours with the array degraded the whole time. The backup only answers durability if you quote its restore time alongside its schedule; a store that has to be restored before anyone can read isn’t available, it is merely not lost. And capacity still ends at whatever the chassis takes, so growing is a forklift rather than a purchase order. Each of those is fixable on its own, and none of them is fixable together on one machine. That, rather than fashion, is the argument for spreading bytes across many of them.
Step 2 · The interface
Buckets and keys, not folders
Before distributing anything, decide the interface. A filesystem with nested directories seems natural — but a global tree that everyone reads and writes becomes a coordination nightmare. What's the right namespace for a store this big?
What namespace should a planet-scale blob store expose?
A hierarchical tree needs locking and atomic rename across a distributed system — expensive coordination that limits scale. Object stores expose a flat namespace precisely to avoid it.
Each object lives in a bucket and is addressed by an opaque key string (which may look like a path but isn't a tree). Flat keys need no directory locking, so the namespace scales to trillions of objects. "Folders" are just a UI convention over key prefixes.
One flat global list has no tenancy or isolation and is impossible to secure or scale per-user. Buckets give ownership and isolation; keys address objects within them.
Expose a flat namespace: buckets (owned containers) holding objects addressed by an opaque key. A key like 2026/07/cat.jpg looks hierarchical but is just a string — there's no real directory tree, no locking, no atomic rename. That flatness is exactly what lets the namespace scale to trillions of objects.
Why this piece earns its place
Flatness buys its scale by quietly assuming two things about how people will use the store: that nobody needs to move objects in bulk, and that every listing anyone wants is prefix-shaped. Both hold until a customer treats keys as folders, which they will, because a key like 2026/07/cat.jpg invites it. Renaming that prefix is not a rename — there is no directory to re-point, so it becomes a copy of every object underneath it followed by a delete of every original, with no single moment at which the operation is done. A client that dies halfway leaves the bucket holding both sets. Deleting a folder has exactly the same shape. The defense is to stop pretending: expose bulk copy and bulk delete as first-class batch operations with their own progress and resumption, and say plainly in the docs that a prefix is a naming convention, not a container. The related habit worth teaching users is to design keys for spread rather than for tidiness, because the prefix everyone writes under is also the part of the key the system will partition on later.
What the new pieces do
- S3 APIgateway
- The HTTP front door. Authenticates, then looks up where an object lives and streams bytes to/from the right storage nodes.
Step 3 · Spread the bytes
A fleet of storage nodes
Capacity beyond one disk means the actual bytes must live across many machines. How should an object's data be laid out over a fleet of storage nodes?
To exceed one disk, where do an object's bytes go?
"Most free space" alone ignores redundancy and can hotspot one node. You need data spread AND replicated for durability, chosen by a placement scheme — not just free-space greed.
One object per node gives no redundancy — that node's failure loses the object. Data must be stored on multiple nodes so any single failure is survivable.
An object is broken into chunks; each chunk is stored on several nodes (or as erasure shards). Capacity scales by adding nodes, and multi-node placement is what makes durability possible.
Run a fleet of commodity storage nodes and split each object into chunks, placing every chunk on several nodes. Capacity now scales by racking more nodes, and — crucially — because each chunk lives on multiple machines, the store can survive any node dying. The nodes are dumb: they hold opaque chunks and know nothing about buckets or keys.
Why this piece earns its place
The knob this step introduces is chunk size, and it is worth setting deliberately rather than inheriting whatever the storage layer defaults to. Small chunks give repair and parallel reads a fine-grained unit: a dead node means rebuilding many small pieces from many sources at once, and one large object can be pulled from several nodes in parallel. But every chunk becomes a row in the index the next step adds, so shrinking the chunk multiplies the size of the tier that is already the hardest part of the system, and it turns one long sequential read into many seeks. Large chunks do the opposite — the index stays small and disks stream — but the repair unit gets coarse, a small object still costs a full chunk’s worth of bookkeeping, and the fleet rebalances in lumps that are awkward to place on a nearly-full node. I’d size the chunk so that disks stream and the index grows with bytes rather than with objects, then handle objects far below that size as their own case rather than bending the chunk size to accommodate them.
What the new pieces do
- Storage Node 1data
- A commodity box full of disks. Holds opaque chunks of object data — it neither knows nor cares what they mean.
- Storage Node 2data
- Another storage box. Objects are spread and replicated across nodes so no single disk failure can lose data.
- Storage Node 3data
- More capacity is just more of these. The data plane scales horizontally to exabytes on cheap disks.
Step 4 · Where does it live?
The metadata service
Bytes are scattered as chunks across thousands of nodes. When a GET for key cat.jpg arrives, how does the system know which nodes hold its chunks — without scanning the whole fleet?
A GET arrives for a key. How do we find which nodes hold its bytes?
Broadcasting to thousands of nodes per request is impossibly wasteful. You need an index that maps a key directly to its chunk locations — the metadata service.
A dedicated index resolves each key to the exact nodes holding its chunks (plus size, checksum, version). One fast lookup, then stream from those nodes. Metadata is separate from data so each scales independently.
Baking locations into the key breaks the moment data is moved, repaired or rebalanced — which happens constantly. Placement must be looked up, not hardcoded.
Add a metadata service: a fast index mapping each bucket/key to its chunk locations, plus size, checksum and version. A request does one metadata lookup to learn where the bytes are, then streams them straight from those storage nodes. Metadata (small, hot, transactional) is kept entirely separate from data (huge, streamed).
Why this piece earns its place
Adding the index puts a lookup in front of every single request, including the reads that used to be nothing but streaming bytes off a disk. That cost is real and it lands unevenly: on a huge download the lookup vanishes into the transfer, while on a small object it can be most of the request. So the first thing I’d do with this design is cache the mapping at the API tier — but cache the right thing. What never changes is a version: the chunk placement of one version of an object is fixed for as long as that version exists, while the key is only a name, and a later PUT re-points it at new bytes. So the cache entry is keyed on bucket, key and version, and an overwrite has to invalidate the key’s current-version entry rather than be left to age out. That leaves two different staleness modes, and only one of them corrects itself. Repair and rebalancing move chunks between nodes constantly, so a cached location is a hint: try it, and a miss or an error sends you back for a fresh lookup. A stale version fails silently instead — it returns the previous object, successfully — which is why the consistency step ahead promises read-after-write for new keys and nothing stronger for overwrites.
- 1 lookupkey → locations
- metasmall & hot
- datahuge & streamed
What the new pieces do
- Metadata Serviceservice
- Maps each bucket/key to the nodes holding its bytes, plus size, checksum and version. The brain that finds the data.
Back of the envelope
- metadata: key → chunks + checksum
- small, transactional, hot
- data: opaque chunks on nodes
- huge, streamed, dumb
- separated
- each tier scales on its own
Step 5 · Never lose a byte
Durability: replication vs erasure coding
"Eleven nines" means that out of ten million objects you'd expect to lose less than one in ten thousand years. Disks fail constantly at this scale. How do you store data so that several simultaneous failures still lose nothing — without paying 3× for every byte?
How do you survive many disk failures without tripling storage cost?
No disk is reliable enough for eleven nines, and one copy dies with its disk. Durability comes from redundancy across independent failure domains, not from better hardware.
3× replication is simple and survives 2 failures, but it triples cost and network. It's used for hot/small data; for cold/large data, erasure coding gives the same durability far cheaper.
Erasure coding splits an object into k data + m parity shards across k+m nodes; any k reconstruct it, surviving m failures at ~1.4× overhead instead of 3×. Replication is simpler and lower-latency for hot data; erasure coding is far cheaper for the vast cold majority. Spread shards across racks/AZs so correlated failures don't take out a whole object.
Use redundancy across independent failure domains. Replication (e.g. 3 copies) is simple and fast — great for hot, small objects. Erasure coding (e.g. 10 data + 4 parity shards; any 10 rebuild the object) survives the same failures at ~1.4× overhead instead of 3×, ideal for the cold majority of data. Spread copies/shards across racks and availability zones so no correlated failure can take out an object.
Why this piece earns its place
Erasure coding earns its saving on the write side and charges for it on the read. A replicated object is served by whichever copy answers first; a 10+4 stripe has to gather ten shards off ten different nodes before a single byte is correct, so the read is only as quick as the slowest of the ten. At fleet scale something is always slow — a disk retrying, a node mid-repair, a switch with a queue — which turns an occasional slow disk into a routinely slow read. The usual defense is to request more shards than are strictly needed and use whichever arrive first, spending a little extra network to cut the tail. The second consequence is a floor on object size: splitting a tiny object into ten data and four parity pieces produces fourteen tiny IOs and fourteen index entries for almost no bytes, and that per-piece overhead can easily exceed what the coding saved. A good part of why small, hot data stays replicated is arithmetic, not just a preference for low latency.
- 11 ninesdurability target
- 3×replication cost
- ~1.4×erasure-coding cost
Back of the envelope
- replication (3×)
- simple, low-latency, hot data
- erasure coding (k+m)
- ~1.4× cost, cold data
- spread across racks/AZs
- no correlated loss
Step 6 · Reads that make sense
Consistency & getting to the bytes
You just uploaded an object and immediately read it back — does it exist yet? And with replicas scattered, which one answers, and how does a request even reach the right region fast?
Provide read-after-write consistency for new objects (a successful PUT is immediately readable) — modern S3 does this by making the metadata commit the point of truth. Front the store with the usual request path: DNS/anycast to the nearest region, a load balancer, then the API tier that resolves metadata and streams from the closest healthy replica. Overwrites and deletes may be eventually consistent across replicas, which clients must expect.
Why this piece earns its place
Anchoring consistency in the metadata commit assumes an ordering the design then has to actually enforce: the chunks are durable on enough nodes to meet the durability target first, and only then does the metadata commit. Get that backwards and a key resolves to bytes that don’t exist yet, which is worse than the object being absent — the store told a client it succeeded and then fails the read. The price of the correct order is orphans. Every upload that dies partway leaves chunks on nodes with no metadata pointing at them, invisible to every code path that starts from a key, so nothing reclaims them by accident. That needs a deliberate sweeper, and the sweeper is the dangerous part: its rule is that an unreferenced chunk can go, which is exactly true of a chunk whose upload is still in flight. So a chunk has to be older than some threshold before it is even eligible — comfortably longer than the slowest upload the system permits — or garbage collection will delete the tail of a write that was about to commit.
Step 7 · Trillions of objects
Scale metadata & huge uploads
One metadata database can't index trillions of objects, and a client can't reliably PUT a 5 TB video in one HTTP request over a flaky network. Two different scaling walls.
Shard the metadata by key (or bucket+key hash) across many nodes, each replicated for durability — so the index scales horizontally with the data. For giant objects use multipart upload: the client splits the object into parts, uploads them in parallel (retrying just the failed parts), and a final "complete" call stitches them into one object. Big transfers become resumable and parallel instead of one fragile stream.
Why this piece earns its place
This step promises that a lookup touches exactly one shard, and that promise quietly assumes every access names an exact key. Sharding on a bucket-plus-key hash is what spreads load evenly, and it is the same property that scatters a single prefix across every shard in the fleet — the keys a customer thinks of as one folder land nowhere near each other. So listing a bucket by prefix is not a small read. It fans out to every shard, each returns whatever it holds under that prefix, and something has to merge those streams into one ordered page. That is a different latency class from a GET and it has to be exposed as one: listing is paginated by an opaque continuation token rather than an offset, because resuming means picking up a merge across every shard, not skipping to a position in a list no single node holds. The consequence worth saying out loud is that the two things a key can be good for pull against each other. Hashing wide is what keeps any one shard from becoming the hot one, and it is exactly what makes listing expensive. If listing is a first-class workload rather than an admin convenience, it earns its own prefix-ordered index instead of a cheaper shard function.
- shardedby key
- multipartparallel + resumable
- 5 TBsingle object
What the new pieces do
- Metadata Storecontrol
- The durable, sharded key-value store behind the metadata service. Sharded by key so it scales to trillions of objects.
Step 8 · The sharp edges
Tiering, integrity & repair
Real object stores must also control cost (most data is cold), guarantee bytes never rot silently, and heal continuously as disks die in the background every single day.
Add lifecycle tiering: automatically move rarely-accessed objects from hot storage to cheaper cold tiers (and eventually archive), by policy. Store a checksum with every object and verify it on read and during background scrubbing to catch silent bit-rot. Run continuous repair: workers detect under-replicated data (from dead nodes) and rebuild copies onto healthy ones. And watch for hot keys — a single viral object — by caching it or fronting the store with a CDN.
Why this piece earns its place
Repair has a throttle, and where you set it is a genuine trade rather than a default to accept. Rebuilding lost redundancy reads from healthy nodes and writes to other healthy nodes, over the same disks and the same network that are serving customer traffic, and it begins at the moment the fleet has just lost capacity. Run it too gently and objects sit under their redundancy target for longer, which is precisely the window in which a second failure is the one that costs you an object — durability is a race between how fast hardware breaks and how fast you heal, and this throttle sets your side of it. Run it too hard and repair browns out reads across the fleet, so one dead node becomes a visible outage for everyone. What makes the trade survivable is priority: rebuild whatever has the fewest surviving pieces first rather than working the backlog in discovery order, and draw the sources for a rebuild from many nodes instead of hammering the few that happen to hold the survivors. Then the throttle can stay modest, because the most exposed data is never queued behind the rest.
What the new pieces do
- Repair / Scrubberrepair
- Background workers that scan for under-replicated or corrupt data and rebuild copies onto healthy nodes.
You did it
You just designed object storage.
Everything you assembled, in order
- One disk caps capacity, durability and availability — spread data across many machines.
- A flat bucket/key namespace (immutable objects) scales where a directory tree can't.
- Split objects into chunks across a dumb, horizontally-scaled storage-node data plane.
- A metadata service maps key → chunk locations — the small, hot, hard part of the system.
- Durability via replication (hot) and erasure coding (~1.4× for cold), spread across racks/AZs.
- Read-after-write consistency for new keys, anchored in the metadata commit.
- Shard metadata by key, multipart-upload huge objects, and tier/scrub/repair continuously.