Design a Distributed Cache — A Guided System Design

Q: What is a distributed cache?

Keep the hot answers in RAM, in front of the database — and spread that RAM across many nodes so it’s bigger than any one machine and survives a node dying. The art is keeping it fast, fresh, and resilient when things fail.

Q: Cache-aside

Use cache-aside: the app asks the cache; on a hit it returns instantly, on a miss it reads the database, then writes the value back into the cache for next time. The app owns the logic; the cache stays a dumb, fast key-value box.

Q: Shard with consistent hashing

Put nodes on a hash ring. Each key maps to the next node clockwise, so adding or removing a node only moves the keys in one arc — a small fraction. The Cache Client hashes the key and talks straight to the owning node.

Q: Eviction & TTL

Give each entry a TTL so it self-expires, and when memory is full, evict by policy: LRU (drop least-recently-used) suits most workloads; LFU favors enduring popularity. The cache becomes a self-managing window onto the hottest data.

Q: Replication & failover

Replicate each shard: a primary serves traffic while one or more replicas stay in sync. On failure, a replica is promoted, so the shard’s keys survive and the database is shielded from a sudden miss storm.

Q: Invalidation

On a write, publish an invalidation message so the relevant cache nodes drop the key (and the next read repopulates it). For writes you can’t afford to be stale, use write-through (update cache and DB together); otherwise lean on TTL + invalidation.

Q: Surviving a thundering herd

Let only one request rebuild a missing key while others wait or serve a slightly stale value: a per-key lock or request coalescing. Add jittered TTLs so keys don’t all expire together, and optionally refresh hot keys before they expire.

Q: The sharp edges

For hot keys, replicate the key across nodes or add a tiny local in-process cache so one shard isn’t crushed. For writes, pick a policy: write-through (consistent, slower) or write-back (fast, risk of loss). Match the policy to how much staleness the data tolerates.

System Design · step by stepDesign a Distributed Cache

Step 1 / 9

▶ Watch it explained

Prefer a video walkthrough?

Design a Distributed Cache — the walkthrough in full

A written version of the interactive walkthrough above — the same steps, decisions and trade-offs, laid out for reading, reference and search.

The big idea

What is a distributed cache?

Your database is the slowest, most precious resource you own, and the same handful of rows get read thousands of times a second. Asking it the same question over and over is both slow for users and an expensive way to melt your database.

Keep the hot answers in RAM, in front of the database — and spread that RAM across many nodes so it’s bigger than any one machine and survives a node dying. The art is keeping it fast, fresh, and resilient when things fail.

How to read this: We add one piece at a time, problem then fix, and the diagram grows into a Redis/Memcached-style cluster. Hit Begin.

Step 1 · The pattern

Cache-aside

The app needs a value. Where does it look first, and what happens when the cache doesn’t have it yet? Get this dance wrong and you either serve stale data or never actually offload the database.

Design decision: The app needs a value that’s usually cached. What’s the read/miss dance?

The call: App checks cache; on a miss, read the DB then backfill the cache. — Cache-aside: a hit returns in <1ms, a miss falls through to the DB and writes the value back for next time. Only requested data gets cached, so memory holds what’s genuinely hot.

Use cache-aside: the app asks the cache; on a hit it returns instantly, on a miss it reads the database, then writes the value back into the cache for next time. The app owns the logic; the cache stays a dumb, fast key-value box.

Lazy population: Only data that’s actually requested gets cached, so memory holds what’s genuinely hot. The first read of a key is slow (a miss); every read after is fast — until it’s evicted or expires.

Step 2 · Bigger than one box

Shard with consistent hashing

One cache node is capped by its RAM and network card. Add nodes naively with hash(key) % n and changing n remaps nearly every key — a mass cache miss that stampedes the database the instant you scale.

Design decision: One node’s RAM is full. You add nodes — how do you map keys so scaling doesn’t stampede the DB?

The call: A consistent-hash ring; key → next node clockwise. — Adding or removing a node moves only the keys in one arc (~1/N), so scaling disturbs a small slice and the database stays shielded from a thundering herd.

Put nodes on a hash ring. Each key maps to the next node clockwise, so adding or removing a node only moves the keys in one arc — a small fraction. The Cache Client hashes the key and talks straight to the owning node.

Why not modulo: Modulo couples every key to the exact node count. Consistent hashing decouples them: membership changes disturb ~1/N of keys, keeping the database safe from a thundering herd of misses during scaling.

Step 3 · Finite memory

Eviction & TTL

RAM is small and the dataset is huge — the cache will fill up. Which entries do you drop to make room, and how do you stop values from lingering forever after they’ve gone stale?

Design decision: RAM is finite and the dataset is huge — the cache will fill. What do you drop, and how do you bound staleness?

The call: TTL per entry + evict by policy (LRU/LFU) under pressure. — TTL bounds staleness; LRU (drop least-recently-used) or LFU (favor enduring popularity) bounds size. The cache becomes a self-managing window onto the hottest data.

Give each entry a TTL so it self-expires, and when memory is full, evict by policy: LRU (drop least-recently-used) suits most workloads; LFU favors enduring popularity. The cache becomes a self-managing window onto the hottest data.

LRU vs LFU vs TTL: TTL bounds staleness; eviction bounds size. LRU is the safe default, but a scan can pollute it — that’s why real systems add scan resistance (e.g. LRU-K, segmented LRU).

Step 4 · Don’t lose a shard

Replication & failover

If a cache node dies, its entire slice of keys instantly becomes misses — and all that traffic slams the database at once. A cache that amplifies failure into a database outage is worse than no cache.

Design decision: A cache node dies and its whole slice of keys becomes misses at once. How do you stop a DB meltdown?

The call: Replicate each shard: primary + replica, promote on failure. — A primary serves while replicas stay in sync; on death a replica is promoted so the shard’s keys survive and the DB is shielded. Async replication is usually fine — losing a few recent sets beats a full-shard blackout.

Replicate each shard: a primary serves traffic while one or more replicas stay in sync. On failure, a replica is promoted, so the shard’s keys survive and the database is shielded from a sudden miss storm.

Availability over strictness: A cache replica needn’t be perfectly consistent — it just needs to absorb load if the primary dies. Async replication is usually fine; losing a few recently-set keys beats a full-shard blackout.

Step 5 · Stay fresh

Invalidation

The moment the database changes, every cached copy of that value is wrong. Serving a stale price or a deleted post erodes trust — but eagerly clearing everything defeats the cache.

Design decision: The DB changes and every cached copy is now wrong. How do you keep the cache fresh?

The call: Publish an invalidation on write so nodes drop the key (+ write-through for critical data). — A write emits an invalidation; the next read repopulates. For data that can’t be stale, write-through updates cache and DB together. TTL remains the safety net if a message is missed.

On a write, publish an invalidation message so the relevant cache nodes drop the key (and the next read repopulates it). For writes you can’t afford to be stale, use write-through (update cache and DB together); otherwise lean on TTL + invalidation.

The hardest problem: “There are only two hard things in computer science…” — cache invalidation is one. Prefer expiring/invalidating keys over trying to keep them perfectly updated; simplicity beats cleverness here.

Step 6 · The stampede

Surviving a thundering herd

A popular key expires. In the same millisecond, a thousand requests all miss, all hit the database for the same value, and all try to recompute it — a cache stampede that can take the database down.

Design decision: A hot key expires and 1,000 requests miss it simultaneously. What stops them all hitting the DB?

The call: Single-flight: one request rebuilds the key, others wait; + jittered TTLs. — A per-key lock / request coalescing ensures N simultaneous misses produce one recompute, not N. Jittering TTLs stops keys expiring together, and hot keys can refresh before expiry.

Let only one request rebuild a missing key while others wait or serve a slightly stale value: a per-key lock or request coalescing. Add jittered TTLs so keys don’t all expire together, and optionally refresh hot keys before they expire.

Coalesce the misses: The fix isn’t a bigger database — it’s ensuring N simultaneous misses produce one recompute, not N. Single-flight plus TTL jitter turns a stampede back into a single quiet miss.

Step 7 · Hot keys & writes

The sharp edges

Consistent hashing assumes keys are roughly equal, but one celebrity key (a viral post) can overwhelm its single owning node. And write-heavy data forces a choice about how cache and database stay in step.

For hot keys, replicate the key across nodes or add a tiny local in-process cache so one shard isn’t crushed. For writes, pick a policy: write-through (consistent, slower) or write-back (fast, risk of loss). Match the policy to how much staleness the data tolerates.

Not all keys are equal: Aggregate metrics hide hotspots. Detect hot keys and treat them specially — local caching, replication, or client-side request batching — rather than assuming uniform load.

You did it

You just designed a distributed cache.

C — a
C — o
T — T
P — e
I — n
S — i
H — o

RUN IT YOURSELF

LRU eviction, in Python & TypeScript

A cache is bounded, so when it fills up it evicts the least-recently-used entry. Here is an O(1) LRU cache in both languages, running live. Switch tabs, read the comments, edit, and hit Run.

HOW TO READ THE CODE — 4 IDEAS

The cache holds at most capacity entries, ordered from oldest to newest.
Every get/put moves the touched key to the most-recent end (steps 1–2).
When it overflows, drop the entry at the oldest end (step 3).
An ordered map (Python OrderedDict / JS Map) makes all of this O(1).

CPython · WebAssembly

# A cache can't hold everything, so it evicts the LEAST-RECENTLY-USED
# entry when full. An ordered dict gives O(1) get/put with LRU order.
from collections import OrderedDict

class LRUCache:
    def __init__(self, capacity):
        self.capacity = capacity
        self.store = OrderedDict()

def get(self, key):
        if key not in self.store:
            return -1
        self.store.move_to_end(key)         # STEP 1 - using it makes it most-recent
        return self.store[key]

def put(self, key, value):
        if key in self.store:
            self.store.move_to_end(key)
        self.store[key] = value             # STEP 2 - newest goes to the recent end
        if len(self.store) > self.capacity:
            self.store.popitem(last=False)  # STEP 3 - over capacity? drop the oldest

cache = LRUCache(2)
cache.put("a", 1); cache.put("b", 2)
cache.get("a")                # touch "a" so "b" becomes the oldest
cache.put("c", 3)             # evicts "b"
print("a:", cache.get("a"), " b:", cache.get("b"), " c:", cache.get("c"))

// A cache can't hold everything, so it evicts the LEAST-RECENTLY-USED
// entry when full. A JS Map preserves insertion order, giving us LRU.
class LRUCache {
  capacity: number;
  store = new Map<string, number>();
  constructor(capacity: number) {
    this.capacity = capacity;
  }

get(key: string): number {
    if (!this.store.has(key)) return -1;
    const value = this.store.get(key)!;
    this.store.delete(key);
    this.store.set(key, value);             // STEP 1 - using it makes it most-recent
    return value;
  }

put(key: string, value: number): void {
    if (this.store.has(key)) this.store.delete(key);
    this.store.set(key, value);             // STEP 2 - newest goes to the recent end
    if (this.store.size > this.capacity) {
      const oldest = this.store.keys().next().value;  // STEP 3 - drop the oldest
      this.store.delete(oldest);
    }
  }
}

const cache = new LRUCache(2);
cache.put("a", 1); cache.put("b", 2);
cache.get("a");                // touch "a" so "b" becomes the oldest
cache.put("c", 3);             // evicts "b"
console.log("a:", cache.get("a"), " b:", cache.get("b"), " c:", cache.get("c"));

Finished this one? 0 / 62 System Designs done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

Distributed Systems Caching & Performance

The concepts behind this design

The same core ideas underpin every system on this site. Master them once, then spot them everywhere.

More System Designs

Cite this page

Reference it in your work, paper or an AI's context window.

APA

Singh, S. (2026). Design a Distributed Cache. Vibe Engines. https://vibeengines.com/systemdesign/distributed-cache-system-design

MLA

Singh, Saurabh. “Design a Distributed Cache.” Vibe Engines, 2026, vibeengines.com/systemdesign/distributed-cache-system-design.

BibTeX

@online{vibeengines-distributed-cache-system-design,
  author       = {Singh, Saurabh},
  title        = {Design a Distributed Cache},
  year         = {2026},
  organization = {Vibe Engines},
  url          = {https://vibeengines.com/systemdesign/distributed-cache-system-design}
}