What is a distributed cache?

Keep the hot answers in RAM, in front of the database — and spread that RAM across many nodes so it’s bigger than any one machine and survives a node dying. The art is keeping it fast, fresh, and resilient when things fail.

Use cache-aside: the app asks the cache; on a hit it returns instantly, on a miss it reads the database, then writes the value back into the cache for next time. The app owns the logic; the cache stays a dumb, fast key-value box.

Shard with consistent hashing

Put nodes on a hash ring. Each key maps to the next node clockwise, so adding or removing a node only moves the keys in one arc — a small fraction. The Cache Client hashes the key and talks straight to the owning node.

Give each entry a TTL so it self-expires, and when memory is full, evict by policy: LRU (drop least-recently-used) suits most workloads; LFU favors enduring popularity. The cache becomes a self-managing window onto the hottest data.

Replication & failover

Replicate each shard: a primary serves traffic while one or more replicas stay in sync. On failure, a replica is promoted, so the shard’s keys survive and the database is shielded from a sudden miss storm.

On a write, publish an invalidation message so the relevant cache nodes drop the key (and the next read repopulates it). For writes you can’t afford to be stale, use write-through (update cache and DB together); otherwise lean on TTL + invalidation.

Surviving a thundering herd

Let only one request rebuild a missing key while others wait or serve a slightly stale value: a per-key lock or request coalescing. Add jittered TTLs so keys don’t all expire together, and optionally refresh hot keys before they expire.

For hot keys, replicate the key across nodes or add a tiny local in-process cache so one shard isn’t crushed. For writes, pick a policy: write-through (consistent, slower) or write-back (fast, risk of loss). Match the policy to how much staleness the data tolerates.

Design a Distributed Cache — A Guided System Design

System Design · step by stepDesign a Distributed Cache

Step 1 / 9

RUN IT YOURSELF

LRU eviction, in Python & TypeScript

A cache is bounded, so when it fills up it evicts the least-recently-used entry. Here is an O(1) LRU cache in both languages, running live. Switch tabs, read the comments, edit, and hit Run.

HOW TO READ THE CODE — 4 IDEAS

The cache holds at most capacity entries, ordered from oldest to newest.
Every get/put moves the touched key to the most-recent end (steps 1–2).
When it overflows, drop the entry at the oldest end (step 3).
An ordered map (Python OrderedDict / JS Map) makes all of this O(1).

CPython · WebAssembly

# A cache can't hold everything, so it evicts the LEAST-RECENTLY-USED
# entry when full. An ordered dict gives O(1) get/put with LRU order.
from collections import OrderedDict

class LRUCache:
    def __init__(self, capacity):
        self.capacity = capacity
        self.store = OrderedDict()

def get(self, key):
        if key not in self.store:
            return -1
        self.store.move_to_end(key)         # STEP 1 - using it makes it most-recent
        return self.store[key]

def put(self, key, value):
        if key in self.store:
            self.store.move_to_end(key)
        self.store[key] = value             # STEP 2 - newest goes to the recent end
        if len(self.store) > self.capacity:
            self.store.popitem(last=False)  # STEP 3 - over capacity? drop the oldest

cache = LRUCache(2)
cache.put("a", 1); cache.put("b", 2)
cache.get("a")                # touch "a" so "b" becomes the oldest
cache.put("c", 3)             # evicts "b"
print("a:", cache.get("a"), " b:", cache.get("b"), " c:", cache.get("c"))

// A cache can't hold everything, so it evicts the LEAST-RECENTLY-USED
// entry when full. A JS Map preserves insertion order, giving us LRU.
class LRUCache {
  capacity: number;
  store = new Map<string, number>();
  constructor(capacity: number) {
    this.capacity = capacity;
  }

get(key: string): number {
    if (!this.store.has(key)) return -1;
    const value = this.store.get(key)!;
    this.store.delete(key);
    this.store.set(key, value);             // STEP 1 - using it makes it most-recent
    return value;
  }

put(key: string, value: number): void {
    if (this.store.has(key)) this.store.delete(key);
    this.store.set(key, value);             // STEP 2 - newest goes to the recent end
    if (this.store.size > this.capacity) {
      const oldest = this.store.keys().next().value;  // STEP 3 - drop the oldest
      this.store.delete(oldest);
    }
  }
}

const cache = new LRUCache(2);
cache.put("a", 1); cache.put("b", 2);
cache.get("a");                // touch "a" so "b" becomes the oldest
cache.put("c", 3);             // evicts "b"
console.log("a:", cache.get("a"), " b:", cache.get("b"), " c:", cache.get("c"));

Finished this one? 0 / 28 System Designs done