Systems & Backend

Thundering Herd

Many clients or workers waking up and retrying at the exact same moment, overwhelming the resource they’re all trying to reach.

A thundering herd happens when a large number of clients are synchronized to act at the same instant — a popular cache key expiring and every request suddenly missing at once, or many clients retrying with the same fixed backoff after an outage clears — and the resulting simultaneous spike overwhelms the very resource (a database, an origin server) that everyone was trying to reach. Jittered backoff (adding randomness to retry timing) and request coalescing are the standard fixes.

Worked example: when a cached item expires or a downed service recovers, thousands of clients hit the origin at the same instant, overwhelming it — the ‘herd’ all stampedes through the one gap. Gotcha: fixes are request coalescing (one caller recomputes, the rest wait on it), jittered/staggered expiry so keys do not all expire together, and randomized backoff on retry; a fleet retrying in lockstep re-creates the herd on every failure.