Databases & Storage
Storing data at scale — transactions, isolation, indexing, ledgers and sync — from the concurrency handbook to systems like Dropbox, a payment ledger and a key-value store.
Handbooks 8
Concurrency, Locks & Isolation Levels
How overlapping operations stay correct — race conditions, optimistic vs pessimistic locking, the four isolation levels and their anomalies, MVCC, and distributed locks with fencing tokens. Part of System Design Fundamentals.
Partitioning, Sharding & Replication
How one dataset becomes many — range vs hash partitioning, consistent hashing and virtual nodes, hot partitions, replication topologies, replication lag, and quorums. Part of System Design Fundamentals.
The SQL Handbook
What the database does underneath your SELECT — the relational model and keys, joins (inner/left/right/full), indexes and B-trees, the query planner and reading EXPLAIN, transactions and ACID, isolation levels and the anomalies they prevent, normalization vs denormalization, and the pitfalls (N+1, missing indexes, SELECT *).
The PostgreSQL Internals Handbook
UPDATE never updates, and a janitor keeps it working. Heap pages and tuples, MVCC's row versions (xmin/xmax visibility arithmetic), VACUUM, bloat and the wraparound scare, B-trees, HOT updates and index-only scans, the WAL's three superpowers (recovery, replication, PITR), reading the planner honestly, and why PgBouncer is infrastructure.
SQL vs NoSQL
Framed as a war, it’s really a menu. Relational databases give you schema, joins and ACID; NoSQL is an umbrella of document/key-value/wide-column/graph stores that drop some of that for flexibility and horizontal scale. The relationships-vs-scale core difference, the CAP trade-off, and why to start relational.
PostgreSQL vs MySQL
Both are excellent free relational databases you can build a company on, so the choice is about character, not capability: Postgres prizes correctness, rich types and advanced features; MySQL prizes simplicity and read-heavy speed. The real differences, when each wins, and why Postgres became the modern default.
OLTP vs OLAP
One storage-layout decision explains the whole comparison: row-oriented OLTP makes fetching one record cheap, column-oriented OLAP makes aggregating one column across millions of rows cheap. Why you can’t run heavy analytics well on your production database.
SQL vs Vector Database
SQL vs vector databases, decided by the kind of question you ask: a relational database answers exact, structured queries over rows (WHERE price < 50); a vector database answers similarity queries over embeddings (find the items most like this). Exact filters vs approximate nearest neighbors, B-tree vs HNSW, pgvector, and why most real systems need both.
Roadmaps 1
System Designs 18
Design YouTube
Build a planet-scale video platform. See how raw uploads become an adaptive-bitrate ladder through an async transcoding pipeline, how a CDN serves immutable segments from the edge, how tiered blob storage holds exabytes affordably, and how view counts stay async.
Design Dropbox
Build a file sync engine. See how splitting metadata from bytes, content-addressed block storage with deduplication, delta sync, a push-then-pull notification service, and conflict-safe versioning fit together to keep files consistent across every device.
Design Ticketmaster
Build an event-booking system. Tackle the double-booking problem head-on: seat inventory, short-lived holds with TTLs, exactly-once purchases via ACID transactions, payment sagas with idempotency, and a virtual waiting room that tames the on-sale stampede.
Design a Key-Value Store
Build a distributed key-value store like DynamoDB. See how consistent hashing places keys, how replication and tunable quorums (R + W > N) trade consistency for availability, how vector clocks resolve conflicts, and how gossip membership, hinted handoff and read repair keep it alive through failure.
Design a Web Crawler
Build a distributed web crawler. See how a URL frontier drives the fetch–parse–enqueue loop, how a bloom-filter seen-set stops infinite re-crawling, how politeness and robots.txt keep you a good citizen, how DNS caching and content dedup remove bottlenecks, and how to shard the frontier by host and dodge traps.
Design Instagram
Build a photo-sharing app. See how splitting media from metadata, a CDN for immutable images, async image processing, a precomputed feed cache, asynchronous like/view counters, TTL-based ephemeral stories, and sharding fit together to serve billions of photos.
Design Pastebin
Build a text-paste service. See how base62 key generation, separating metadata from blob storage, a cache and CDN for the read-heavy immutable path, TTL expiration, privacy flags and size limits, and async analytics fit together.
Design a Search Engine
Build a web search engine. See how an inverted index turns search into list lookups, how query parsing mirrors indexing, how an offline indexer builds postings, how BM25 and PageRank rank results, and how document-sharded scatter-gather, result caching and a fresh index scale it to billions of pages.
Design Airbnb
Build a lodging marketplace. See how listings and geo search with filters, per-listing availability calendars, atomic booking that never double-books, payment hold-and-capture with idempotency, event-driven reviews and reindexing, and read replicas fit together.
Design a Payment System
Build a payment system like Stripe. See how a payment is modeled as a state machine of authorize and capture, how idempotency keys prevent double-charges, how a double-entry ledger keeps every cent auditable, how tokenization shrinks PCI scope, and how async webhooks, reconciliation and a durable retryable worker handle slow banks and failure.
Design Object Storage
Build an object store like Amazon S3. See why one disk caps capacity, durability and availability, how a flat bucket/key namespace scales to trillions of immutable objects, how a metadata service maps keys to chunk locations, how replication and erasure coding deliver eleven-nines durability, how read-after-write consistency is anchored in metadata, and how multipart uploads, tiering and background repair work.
Design a Distributed File System
Build a distributed file system like GFS or HDFS. See why one file server caps capacity and throughput, how huge files split into large chunks across chunkservers, how a single master serves only metadata while clients stream bytes directly, how rack-aware replication survives correlated failures, how a leased primary orders pipelined writes, and how the master is made recoverable.
Design a Time-Series Database
Build a time-series database like InfluxDB or Prometheus TSDB. See why a row-per-point SQL table melts, how series = metric + tags models the data, how columnar storage with delta-of-delta timestamps and XOR values shrinks points to ~1–2 bytes, how an LSM write path with a WAL swallows the firehose, how sharding by series and downsampling work, and why cardinality is the real scaling limit.
Design a Log Search System
Build a log aggregation and search system like the ELK stack or Loki. See why grep-over-SSH can't scale, how structured logs and shipper agents get data off ephemeral hosts, how a Kafka buffer absorbs bursts, how a parse/enrich pipeline feeds an inverted index, how time-based sharding and scatter-gather search billions of lines in milliseconds, and how hot-warm-cold retention bounds cost.
Design a Digital Wallet
Build a digital wallet like PayPal or Venmo. See why a mutable balance column is dangerous, how an append-only double-entry ledger makes balance derived and money conserved, how atomic debit+credit transfers survive a mid-transfer crash, how idempotency keys stop double-charges, how per-account locking prevents double-spend, how cross-shard transfers use sagas, and how pending/settled states and reconciliation handle slow bank rails.
Design a Proximity Service
Build a proximity / nearby-search service like Yelp or "find drivers near me". See why SQL distance queries degrade to full scans, the 2D indexing problem, how geohash encodes 2D into a locality-preserving 1D key, how quadtrees and S2 cells adapt to density, the cell-boundary problem, caching hot cells, and how moving objects and geo-sharding are handled.
Design a Logging Pipeline
Build the ingestion pipeline that collects, buffers and delivers logs from thousands of services — the write path, not the query/search side. See why synchronous writes to a central log server are dangerous, local agent buffering, bounded backpressure and overflow policy, batching for throughput, where structured parsing should happen, at-least-once delivery with dedup, sampling at high volume, and why a logging storm can take down the very system it was meant to observe.
Design an Email System
Build an email system like Gmail or Outlook. See why raw inbound mail can't land straight in a mailbox, retry-with-backoff for unreliable delivery across the open internet, layered spam/authentication filtering, per-user mailbox sharding, thread reconstruction via message headers instead of subject lines, delivery dedup and bounce-loop prevention, and IMAP-style incremental multi-device sync.
Algorithm Games 1
Labs 2
The Query Planner
Don't guess why your query is slow — watch the planner choose. A cost-based query planner estimates the cost of every way to run a query — full table scan, index scan, bitmap AND of two indexes — and picks the cheapest. The catch: an index isn't always a win. Change how selective each predicate is, add or drop indexes, and watch the planner flip between a sequential scan and an index scan for reasons you can finally see. Made playable, with theory and a quiz.
B-Tree vs LSM-Tree
Don't read about storage engines — race them. A B-tree updates data in place: reads are cheap and stable, but every write is a random disk write. An LSM-tree appends writes to an in-memory buffer, flushes them as sorted runs, and compacts in the background: writes are cheap and sequential, but a read may have to check several runs. Stream the same writes into both, watch flushes and compaction happen, and see which engine wins for writes and which for reads — made playable, with theory and a quiz.
Interactive Tools 2
Database Index Size Estimator
A database index size estimator. Enter the row count, the size of the indexed key, and a fill factor to estimate the on-disk size of a B-tree index — including the row-pointer overhead and internal nodes — so you can anticipate the storage and memory cost of an index before adding it. Compares the cost of several indexes at once.
Storage Growth Forecaster
A storage growth forecaster. Enter your current data size and either a daily ingest rate or a monthly growth percentage, set a replication factor, and see your projected size at 3, 6, 12 and 24 months — plus when you would cross a capacity threshold. Handy for capacity planning databases, logs, object stores and backups.
About databases & storage
Storing data at scale is where correctness and performance collide. Transactions and isolation levels decide what "correct" means under concurrency; indexing decides whether a query is instant or hopeless; replication and sharding decide how you grow past one machine without losing data or your mind.
This topic spans the concurrency handbook and the systems that live or die on storage decisions — a payment ledger that must never double-charge, a key-value store, a sync engine like Dropbox. The through-line: the database is usually the hardest part of a system to change later, so its guarantees are worth understanding up front.