The big idea

What is object storage?

You need to store an effectively unlimited number of files — photos, videos, backups, logs — of any size, reachable from anywhere, that you can never lose. A folder on a server's disk fails on all three: it fills up, it dies with the disk, and it's unreachable when the box is down.

Object storage (S3, GCS, Azure Blob) stores blobs by key behind a simple HTTP API: PUT to store, GET to read. Behind that flat interface it spreads and replicates every object across thousands of machines to deliver near-infinite capacity and famous "eleven nines" of durability (99.999999999%).

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the diagram grow, and at the end lose a node and stall metadata to see what durability and scale really cost. Hit Begin.

Step 1 · The naive store

Files on one disk

Start with the obvious: save each upload as a file on one server's disk. It works for a demo. Name the three ways it fails as you grow.

Design decision: Files on a single server's disk. What are the fundamental limits?

The call: Capacity, durability and availability all cap at one machine. — One disk fills up; when it dies your data dies with it; when the box is down nobody can read. Object storage exists to break all three ceilings by spreading data across many machines.

One disk means one ceiling on capacity, one point of data loss, and one point of unavailability. To store unlimited data you can't lose and can always reach, the bytes must live on many machines. Everything else follows from that.

The three properties: Object storage is really a machine for delivering three things at once: near-infinite capacity (add nodes), extreme durability (redundancy so failures don't lose data), and high availability (always reachable). No single disk can offer any of them at scale.

Step 2 · The interface

Buckets and keys, not folders

Before distributing anything, decide the interface. A filesystem with nested directories seems natural — but a global tree that everyone reads and writes becomes a coordination nightmare. What's the right namespace for a store this big?

Design decision: What namespace should a planet-scale blob store expose?

The call: A flat namespace: buckets containing objects addressed by key. — Each object lives in a bucket and is addressed by an opaque key string (which may look like a path but isn't a tree). Flat keys need no directory locking, so the namespace scales to trillions of objects. "Folders" are just a UI convention over key prefixes.

Expose a flat namespace: buckets (owned containers) holding objects addressed by an opaque key. A key like 2026/07/cat.jpg looks hierarchical but is just a string — there's no real directory tree, no locking, no atomic rename. That flatness is exactly what lets the namespace scale to trillions of objects.

Immutable objects: Objects are typically immutable: you replace an object wholesale (a new version), you don't edit bytes in place. Immutability makes replication, caching and consistency dramatically simpler — a stored object never changes under you.

Step 3 · Spread the bytes

A fleet of storage nodes

Capacity beyond one disk means the actual bytes must live across many machines. How should an object's data be laid out over a fleet of storage nodes?

Design decision: To exceed one disk, where do an object's bytes go?

The call: Split into chunks, placed and replicated across many storage nodes. — An object is broken into chunks; each chunk is stored on several nodes (or as erasure shards). Capacity scales by adding nodes, and multi-node placement is what makes durability possible.

Run a fleet of commodity storage nodes and split each object into chunks, placing every chunk on several nodes. Capacity now scales by racking more nodes, and — crucially — because each chunk lives on multiple machines, the store can survive any node dying. The nodes are dumb: they hold opaque chunks and know nothing about buckets or keys.

Data plane vs control plane: The data plane (storage nodes) just stores and serves bytes, fast and simple. The control plane (next step) decides where bytes go and how to find them. Separating them lets each scale on its own — a recurring pattern in big storage systems.

Step 4 · Where does it live?

The metadata service

Bytes are scattered as chunks across thousands of nodes. When a GET for key cat.jpg arrives, how does the system know which nodes hold its chunks — without scanning the whole fleet?

Design decision: A GET arrives for a key. How do we find which nodes hold its bytes?

The call: A metadata service maps key → chunk locations, size, checksum, version. — A dedicated index resolves each key to the exact nodes holding its chunks (plus size, checksum, version). One fast lookup, then stream from those nodes. Metadata is separate from data so each scales independently.

Add a metadata service: a fast index mapping each bucket/key to its chunk locations, plus size, checksum and version. A request does one metadata lookup to learn where the bytes are, then streams them straight from those storage nodes. Metadata (small, hot, transactional) is kept entirely separate from data (huge, streamed).

Metadata is the hard part: Storing petabytes on cheap disks is almost easy. The genuinely hard problem is the metadata: a consistent, low-latency index over trillions of objects that must never lie about where the bytes are. That's why it gets its own sharded, replicated tier.

Step 5 · Never lose a byte

Durability: replication vs erasure coding

"Eleven nines" means that out of ten million objects you'd expect to lose less than one in ten thousand years. Disks fail constantly at this scale. How do you store data so that several simultaneous failures still lose nothing — without paying 3× for every byte?

Design decision: How do you survive many disk failures without tripling storage cost?

The call: Replicas for hot data; erasure coding (e.g. 10+4) for the rest. — Erasure coding splits an object into k data + m parity shards across k+m nodes; any k reconstruct it, surviving m failures at ~1.4× overhead instead of 3×. Replication is simpler and lower-latency for hot data; erasure coding is far cheaper for the vast cold majority. Spread shards across racks/AZs so correlated failures don't take out a whole object.

Use redundancy across independent failure domains. Replication (e.g. 3 copies) is simple and fast — great for hot, small objects. Erasure coding (e.g. 10 data + 4 parity shards; any 10 rebuild the object) survives the same failures at ~1.4× overhead instead of 3×, ideal for the cold majority of data. Spread copies/shards across racks and availability zones so no correlated failure can take out an object.

Erasure coding, briefly: Split an object into k data shards and compute m parity shards (Reed–Solomon). Store all k+m on different nodes; any k of them reconstruct the original, so you tolerate m losses. It's the math behind cheap, extreme durability — the same idea as RAID, generalized across machines.

Step 6 · Reads that make sense

Consistency & getting to the bytes

You just uploaded an object and immediately read it back — does it exist yet? And with replicas scattered, which one answers, and how does a request even reach the right region fast?

Provide read-after-write consistency for new objects (a successful PUT is immediately readable) — modern S3 does this by making the metadata commit the point of truth. Front the store with the usual request path: DNS/anycast to the nearest region, a load balancer, then the API tier that resolves metadata and streams from the closest healthy replica. Overwrites and deletes may be eventually consistent across replicas, which clients must expect.

Consistency is a metadata property: Because data is immutable, "does this object exist and where?" is answered entirely by the metadata. Make the metadata write the atomic commit and read-after-write falls out for new keys. The scattered data replicas only need to converge eventually.

Step 7 · Trillions of objects

Scale metadata & huge uploads

One metadata database can't index trillions of objects, and a client can't reliably PUT a 5 TB video in one HTTP request over a flaky network. Two different scaling walls.

Shard the metadata by key (or bucket+key hash) across many nodes, each replicated for durability — so the index scales horizontally with the data. For giant objects use multipart upload: the client splits the object into parts, uploads them in parallel (retrying just the failed parts), and a final "complete" call stitches them into one object. Big transfers become resumable and parallel instead of one fragile stream.

Sharded metadata: The metadata tier is itself a sharded, replicated distributed database — often the most sophisticated part of the whole system. Partition by key so lookups hit exactly one shard, and replicate each shard (via consensus) so a failed leader doesn't lose the index.

Step 8 · The sharp edges

Tiering, integrity & repair

Real object stores must also control cost (most data is cold), guarantee bytes never rot silently, and heal continuously as disks die in the background every single day.

Add lifecycle tiering: automatically move rarely-accessed objects from hot storage to cheaper cold tiers (and eventually archive), by policy. Store a checksum with every object and verify it on read and during background scrubbing to catch silent bit-rot. Run continuous repair: workers detect under-replicated data (from dead nodes) and rebuild copies onto healthy ones. And watch for hot keys — a single viral object — by caching it or fronting the store with a CDN.

Design for the unhappy path: Cold data → tier it down. Bit-rot → checksums + scrubbing. Dead node → background re-replication. Viral object → cache/CDN in front. At exabyte scale, hardware is always failing somewhere; the system is designed to heal faster than it breaks.

You did it

You just designed object storage.

• O — n
• A —
• S — p
• A —
• D — u
• R — e
• S — h