Keep the answer
close.

A cache sits between your application and a slow source of truth — a database, an external API, a disk — and keeps a copy of recently or frequently used data in a fast store, usually memory. When a request arrives, one of two things happens. Either the data is in the cache, a hit, and you return it in microseconds; or it isn't, a miss, and you pay the full cost of the slow source, then typically store the result so the next request for that key hits.

That's the entire mechanism, and it's a bet: you're wagering that data fetched once will be wanted again soon, so keeping it close pays off. The payoff isn't guaranteed — a cache full of things nobody re-requests is just wasted memory — which is why the fraction of hits, not the raw speed of the cache, is what actually determines whether caching was worth it. The next section makes that precise.

Suppose a fraction h of requests hit the cache and cost t_cache, while the rest miss and cost t_source. The average latency is just the weighted average: h·t_cache + (1−h)·t_source. Because the source is usually far slower than the cache — a millisecond of memory versus a hundred milliseconds of database — the miss term dominates. Almost all of your average latency is the tax you pay on the requests that missed.

That's why hit rate is the metric that matters. Push it from 50% to 90% and you've cut the number of expensive misses fivefold, dragging average latency down toward the cache's speed. Let it slip toward zero and the cache does nothing but add a lookup before every slow fetch. This is also why a small increase in hit rate at the high end is so valuable: going from 95% to 99% cuts the miss traffic — and the load on your source — by five times. The whole discipline of caching is a fight to raise that one number without serving stale data.

How data gets into and out of the cache defines the pattern, and each trades consistency, write speed, and durability differently:

Most systems use cache-aside for reads: it's simple, and the cache only holds what's actually been requested. The write patterns are about keeping the copy honest — write-through pays latency for consistency, write-back pays a durability risk for speed. There's no universally right answer; you pick based on whether stale reads or slow writes or lost writes hurt your workload the most. And underneath all of them sits an eviction policy — usually LRU, evict the least-recently-used — because the cache is finite and something has to leave when it fills.

The average request latency is the hit-rate-weighted blend of the two paths, and it's dominated by the miss term because the source is slow:

h=0.9, t_cache=1 ms, t_source=100 ms → 0.9·1 + 0.1·100 = 10.9 ms. At h=0 it's the full 100 ms; the cache earns its keep entirely through h.

Rearranging tells you the hit rate you'd need to meet a latency target, and the speedup over always hitting the source:

To get avg ≤ 20 ms with those numbers you need h ≥ (100−20)/(100−1) = 80/99 ≈ 0.81. The runnable version below computes hit rate, effective latency, speedup, and the required hit rate.

There's an old joke: there are only two hard things in computer science — cache invalidation and naming things. The first is real and it's the deep problem of caching. A cache holds a copy, and the moment the source changes, that copy is stale — serving old data until it expires or is explicitly removed. You manage staleness with a TTL (time-to-live: expire the entry after N seconds) and with explicit invalidation on writes (delete or update the cached key when the source changes). Both are trade-offs: a short TTL keeps data fresh but lowers hit rate; a long TTL is fast but risks serving stale answers. There's no free lunch — you're always choosing where on the freshness-versus-hit-rate curve to sit.

The second classic is the cache stampede (thundering herd). A single popular key expires, and in that instant every in-flight request for it misses at once — and all of them hit the slow source simultaneously, potentially overwhelming the very database the cache was protecting. The fixes are about coordination: a lock so only one request refills while others wait, request coalescing so duplicate misses share one fetch, or early/staggered expiry so hot keys refresh before they expire and never all fall due at once. Both problems come from the same root — a cache is a copy that can be wrong or can vanish — and handling them is what separates a cache that speeds things up from one that occasionally takes the whole system down.

The first mistake is caching without measuring the hit rate. A cache with a 10% hit rate is adding a lookup before nearly every slow fetch — negative value — and you won't know unless you instrument hits and misses. The second is caching things that change constantly: if a key is written as often as it's read, the cache is stale most of the time and you pay invalidation cost for almost no benefit. Cache what's read far more than written.

Two more. Unbounded caches: memory is finite, so a cache with no eviction policy and no size limit eventually consumes all of it and crashes the process — always set a max size and an eviction policy (LRU is the sane default). And the cold-start / first-request miss: cache-aside means the very first request for any key is always a miss, so a freshly deployed or restarted cache has a 0% hit rate and offers no protection exactly when a cold system is most fragile — warm critical caches ahead of traffic if you can. Get the hit rate high, the invalidation honest, and the size bounded, and a cache turns a slow, overloaded system into a fast, calm one. The whole handbook is one sentence: latency is the blend, hit rate is the dial, and the copy must stay correct.