The big idea
How does Dropbox sync your files?
You drop a file in a folder on your laptop. Moments later it’s on your phone, your tablet, and a colleague’s machine — and if you tweak one paragraph, only that change travels, not the whole file again.
Sync is really two problems wearing one coat: moving bytes efficiently, and keeping a consistent view of the namespace (the file tree, names, versions) across devices that come and go. We’ll build those two halves and the machinery that keeps them agreeing.
How to read this: Each step opens with a real design decision — you make the call before I show you what ships. Watch the diagram grow into a sync engine, and at the end cut the notifier to see push degrade to poll. Hit Begin.
Step 1 · The skeleton
Files = metadata + bytes
A "file" is two things: where it lives and what it’s called (metadata), and the actual content. Conflating them makes both sync and storage harder.
Design decision: A "file" is its name/path (metadata) plus the actual bytes (content). How do you model it for sync?
The call: Split: a Metadata Service owns the file tree; bytes live separately. — A Metadata DB owns folders, names and versions; content goes elsewhere. Devices sync the small metadata first to learn exactly what changed, then fetch only the bytes they’re missing. The file tree, not the files, is kept consistent.
Split them. A Metadata Service (backed by a Metadata DB) owns the folder tree, names, and versions. Content goes elsewhere. Uploading a file becomes: store the bytes, then record a row saying "this path, this version, is made of these bytes."
The metadata is the truth: Devices sync the metadata first — it’s small and tells each client exactly what changed. Only then do they fetch the bytes they’re missing. The file tree, not the files, is what’s kept consistent.
Step 2 · Chunk & dedup
Don’t move the whole file
Re-uploading a 2 GB file because you changed one slide is absurd — and storing the same attachment a thousand users share a thousand times is wasteful.
Design decision: Re-uploading a 2 GB file for a one-slide change is absurd, and storing a shared attachment 1000× is wasteful. Fix?
The call: Split files into fixed-size blocks, hash each, store by hash (dedup). — A file becomes an ordered list of block hashes; a block named by its content hash is stored once globally, so if its hash already exists you skip the upload. Identical data — across files and users — is stored a single time.
The Block Service splits every file into fixed-size blocks (say 4 MB), hashes each, and stores blocks in a content-addressed Block Store. A file’s metadata becomes an ordered list of block hashes. If a block’s hash already exists, you skip the upload entirely — deduplication.
Content-addressed blocks: Naming a block by the hash of its contents means identical data is stored once, globally. Edit a file and only the blocks that actually changed are new — the rest are already there.
Step 3 · Delta sync
Send only what changed
Even within one file, you usually touch a tiny fraction. Sending every block on every save wastes bandwidth and battery.
Design decision: Even within one file you usually touch a tiny fraction. How do you avoid sending every block on every save?
The call: Re-chunk, diff new block hashes against the old list, upload only new blocks. — Because a file is a list of content-addressed blocks, "what changed" is a set difference of hashes — delta sync. The network only ever carries genuinely new bytes; metadata records a new version pointing mostly at existing blocks.
On a change, the client re-chunks the file, compares the new block hashes against the old list, and uploads only the new blocks. The metadata service records a new version pointing mostly at existing blocks plus a few new ones. Downloads work the same in reverse — fetch only the blocks you don’t already have.
Diff in block-space: Because files are just lists of content-addressed blocks, computing "what changed" is a set difference of hashes. That’s delta sync: the network only ever carries genuinely new bytes.
Step 4 · Multi-device
Tell the other devices, now
The laptop uploaded a change — but the phone has no idea. Polling "anything new?" every few seconds is laggy and hammers your servers with millions of empty checks.
Design decision: The laptop uploaded a change but the phone has no idea. How do other devices learn quickly without hammering your servers?
The call: A Notification Service holds a connection per device; push a nudge on change. — On commit, the metadata service emits a sync event; the notifier pushes "something changed" to affected online devices, which pull the new metadata and missing blocks. Push (light) then pull (heavy) — and via the event stream the writer never needs to know who’s online.
Add a Notification Service that holds a long-lived connection (or long-poll) to every online device. When the metadata service commits a change, it emits a sync event; the notifier pushes a nudge to the affected devices, which then pull the new metadata and missing blocks.
Push, then pull: A lightweight push ("something changed") wakes the right devices instantly; the heavier pull (metadata + blocks) happens on demand. Decoupling them via an event stream means the writer never needs to know who’s online.
Step 5 · Serve downloads
Popular files, everywhere
When a shared file goes viral inside a company, thousands of devices pull the same blocks from the origin block store at once — slow for far-away users and heavy on the origin.
Design decision: A shared file goes viral — thousands of devices pull the same blocks from origin at once. How do you serve downloads?
The call: Put a CDN/cache in front; immutable content-addressed blocks cache forever. — A block named by its hash can never change, so it caches anywhere with zero invalidation. Downloads fetch hot blocks from the nearest edge; origin only serves misses and the long tail — downloads scale almost for free.
Put a CDN / cache in front of the block store. Blocks are immutable and content-addressed — a perfect cache target. Downloads fetch hot blocks from the nearest edge; the origin only serves cache misses and the long tail.
Immutable blocks love caches: A block named by its hash can never change, so it can be cached forever, anywhere, with zero invalidation logic. That’s what makes downloads scale almost for free.
Step 6 · Scale the metadata
Billions of files
Block storage scales easily, but the metadata — every file, folder, and version for hundreds of millions of users — outgrows a single database.
Design decision: Block storage scales easily, but metadata for 100s of millions of users outgrows one DB. How do you partition it?
The call: Shard the metadata DB by user (or workspace). — A user’s files are accessed together and rarely cross into another user’s tree, so user id is a clean shard key: even load, operations local to one shard, no cross-shard transactions for everyday ops. Read replicas handle listing-heavy load.
Shard the metadata DB by user (or workspace), so all of one account’s tree lives together and operations stay local to a shard. Add read replicas for listing-heavy workloads. The block store, already object storage, scales horizontally on its own.
Shard by the natural boundary: A user’s files are almost always accessed together and rarely cross into another user’s tree, so user id is a clean shard key: even load, no cross-shard transactions for everyday operations.
Step 7 · The sharp edges
Conflicts, offline & sharing
Two devices edit the same file while one is offline. Whose version wins? And how do permissions work when a folder is shared with a team?
Track versions and detect divergence; rather than silently losing data, keep both as a conflicted copy and let the user reconcile. Queue changes made offline and replay them on reconnect. Layer sharing and permissions onto the metadata tree so access is checked on every operation.
Never lose a byte: The cardinal rule of sync is that no edit silently disappears. When you can’t auto-merge, preserve both versions and surface the conflict — predictable and safe beats clever and lossy.
You did it
You just designed Dropbox.
- Split metadata (the file tree) from content (the bytes).
- Chunk files into content-addressed blocks and deduplicate.
- Delta sync moves only the blocks that actually changed.
- A notification service pushes change events; devices then pull.
- A CDN caches immutable hot blocks for fast downloads everywhere.
- Shard the metadata DB by user to scale to billions of files.
- Versioning, conflict copies, offline queues and sharing for real life.
