Vibe Engines
YouTube
SYSTEMS & BACKEND

Caching & Performance

Making systems fast — caching strategies, CDNs, rate limiting and low-latency design — with a capacity estimator and the systems that live or die on cache hit rates.

39 pieces · 6 formats Follow this topic (RSS)

Caching means keeping the result of an expensive operation somewhere fast so the next request can skip the work. A database query that takes milliseconds becomes a sub-millisecond memory read; an image served from a location near the user avoids a cross-continent round trip. Performance engineering is the wider discipline of finding where the time goes on a request’s path and removing the slowest part first.

The ideas that decide the outcome

  • Hit rate is the number that matters. Average latency is roughly the hit rate times cache latency plus the miss rate times origin latency, and the miss term usually dominates. Raising the hit rate from 90% to 99% cuts misses tenfold.
  • Choose a write pattern deliberately. Cache-aside loads data on a miss and is the common default; write-through updates the cache on every write; write-back acknowledges writes before they reach the database, trading durability risk for speed.
  • Eviction is a bet about the future. LRU bets that recently used items will be used again; LFU bets on consistently popular ones. A single large scan can flush an LRU cache of everything that mattered.
  • Invalidation is the hard part. Every cache serves stale data for some window. TTLs bound the window, explicit invalidation shortens it, and versioned keys or URLs avoid it for content that never changes in place.
  • Measure the tail, not the mean. Users feel p95 and p99 latency, and one slow dependency in a fan-out makes the whole request slow in a way an average hides.

How the formats fit together

Start with the latency numbers explorer to feel the gap between memory, disk and network, because every caching decision follows from it. The handbooks cover caching patterns, Redis as a data-structure server and the Redis-or-Memcached choice. The labs race eviction policies on one access stream and show what a bounded queue does under overload, and the coding challenges have you build an LRU cache, rate limiters, a Bloom filter and a streaming median. The system designs apply it all at scale in CDNs, distributed caches, rate limiters, feeds, flash sales and an LLM cache, while the calculators size capacity, model cache savings and compute percentiles from real numbers.

Common mistakes

  • Adding a cache before measuring where the latency actually is.
  • Letting a popular key expire under load, so many requests miss at once and stampede the database.
  • Caching without a plan for invalidation, then debugging stale data in production.
  • Reporting average latency and missing the slow tail users complain about.

Start here, in this order

  1. Latency Numbers Explorer — Feel the gaps between a memory read, an SSD read and a network round trip, scaled to human time.
  2. The Caching Patterns Handbook — Learn the hit-rate math, the three write patterns and the two problems that turn a cache into an outage, staleness and stampedes.
  3. The Eviction Race: LRU vs LFU — Run FIFO, LRU and LFU on the same access stream and watch their hit rates diverge on a hot-key-plus-scan workload.
  4. LRU Cache — Build the eviction policy behind most size-limited caches, with O(1) get and put.
  5. Design a Distributed Cache — Scale the cache across machines with consistent hashing, replication, TTLs and defenses against hot keys.
  6. Percentile Calculator — Turn raw latencies into p50 through p99 and see why the average hides the requests users notice.

Questions

When should I add a cache?

Add a cache when measurement shows that a specific operation is slow or expensive, it is read far more often than it changes, and serving slightly stale data for a bounded time is acceptable. Typical candidates are database reads for popular items, results of costly computations, rendered pages and responses from slow third-party APIs. First check whether an index, a better query or batching solves the problem more simply, because a cache adds a second copy of the data that you now have to keep correct, monitor and size.

What is the difference between LRU and LFU eviction?

Both decide what to remove when the cache is full. LRU, least recently used, evicts the item that has gone longest without being accessed, which suits workloads where recent items tend to be reused. LFU, least frequently used, evicts the item with the fewest accesses, which protects consistently popular items from being pushed out by a burst of one-time reads such as a full scan. LRU is simpler and the more common default; LFU needs extra bookkeeping and a way to age old counts so yesterday’s favorites eventually leave.

What is a cache stampede, and how do you prevent it?

A cache stampede happens when a popular entry expires or is evicted and many concurrent requests miss at the same moment, all recomputing the value or querying the database at once. The sudden load can slow or take down the very system the cache was protecting. Common defenses are letting only one request rebuild the value while the others wait or receive the stale copy, refreshing hot entries before they expire, adding random jitter to TTLs so keys do not expire together, and serving stale data while revalidating in the background.

Should I use Redis or Memcached?

Memcached is a simple, multithreaded in-memory key-value cache that stores opaque values and treats its data as disposable, which makes it a good fit when all you need is a fast cache that uses many CPU cores easily. Redis is an in-memory data-structure server with hashes, lists, sets, sorted sets and streams, plus optional persistence, replication and clustering, so it also handles leaderboards, rate limiters, queues and sessions. Most new projects choose Redis for that flexibility; Memcached remains a sound choice for plain, high-throughput caching.

Handbooks 3

Handbook

The Redis Handbook

Redis as an in-memory data-structure server, not just a cache — the core structures (strings, hashes, lists, sets, sorted sets), why single-threaded is fast, RDB vs AOF persistence, eviction and TTL, the classic patterns (cache, rate limiter, leaderboard, session, queue, pub/sub), high availability with replication, Sentinel and Cluster, and when it's the wrong tool.

Engineering
Handbook

Redis vs Memcached

Both put data in RAM by a key, so they look interchangeable — but Memcached does one thing (a lean multithreaded cache) while Redis is a data-structure server with persistence, replication and pub/sub. The one-job-vs-many split, a feature table, and why most teams now default to Redis.

EngineeringComparison
Handbook

The Caching Patterns Handbook

Why the fastest query is the one you never run. A cache serves requests from a fast store and falls back to a slow source on a miss, so average latency is a hit-rate-weighted blend of the two paths — h·t_cache + (1−h)·t_source, dominated by the miss term. That makes hit rate the one number that decides a cache's value. The patterns (cache-aside, write-through, write-back), the eviction policy, and the two famously hard problems — invalidation/staleness and the cache stampede — that turn a speedup into an outage if you get them wrong. With worked math and runnable code.

Engineering

System Designs 19

System Design

Design a URL Shortener

Build a URL shortener (think Bitly or TinyURL). Learn how to mint unique short codes with base62, make the read-heavy redirect path sub-millisecond with caching, shard billions of mappings, and track clicks asynchronously.

CachingShardingScalability
System Design

Design Twitter

Build Twitter's news feed. Learn the social graph, why fan-out on write beats read-time merging, how a precomputed timeline cache makes feed reads O(1), how ranking surfaces the best tweets, and how a hybrid model solves the celebrity problem.

Fan-outCachingScalability
System Design

Design YouTube

Build a planet-scale video platform. See how raw uploads become an adaptive-bitrate ladder through an async transcoding pipeline, how a CDN serves immutable segments from the edge, how tiered blob storage holds exabytes affordably, and how view counts stay async.

StorageCDNScalability
System Design

Design a Rate Limiter

Build a distributed rate limiter. Learn where to put the check, how to key and tier limits, the token-bucket and sliding-window algorithms, why shared atomic counters in Redis avoid race conditions, and the fail-open vs fail-closed trade-off.

Distributed SystemsCachingReliability
System Design

Design Typeahead (Autocomplete)

Build a search typeahead. See how a prefix trie turns search into an O(L) walk, how a prefix-keyed cache answers the skewed common case in under a millisecond, how precomputed top-k ranking surfaces the best ten, how an offline builder keeps suggestions fresh from query logs, and how to personalize and shard.

TriesCachingRanking
System Design

Design a Distributed Cache

Build a distributed cache like Redis or Memcached. See how cache-aside shields the database, how consistent hashing shards across nodes, how TTL and LRU/LFU eviction bound memory, how replication and invalidation keep it correct, and how to survive cache stampedes and hot keys.

CachingShardingReliability
System Design

Design a News Feed

Build a social news feed. See how the social graph, pull vs push fan-out, a precomputed feed cache, relevance ranking, the hybrid model that tames the celebrity problem, blending multiple content sources, and cursor pagination fit together.

Fan-outCachingRanking
System Design

Design Netflix

Build a planet-scale video streaming service. See how the play path splits authorization from byte delivery, how origin storage and CDN edges serve immutable segments, how an adaptive-bitrate ladder adapts to any connection, how a parallel transcoding pipeline builds it, and how Open Connect, recommendations and QoE events fit together.

StreamingCDNScalability
System Design

Design Instagram

Build a photo-sharing app. See how splitting media from metadata, a CDN for immutable images, async image processing, a precomputed feed cache, asynchronous like/view counters, TTL-based ephemeral stories, and sharding fit together to serve billions of photos.

StorageCDNFan-out
System Design

Design Pastebin

Build a text-paste service. See how base62 key generation, separating metadata from blob storage, a cache and CDN for the read-heavy immutable path, TTL expiration, privacy flags and size limits, and async analytics fit together.

StorageCachingScalability
System Design

Design a Stock Exchange

Build a stock exchange matching engine. See how an in-memory order book with price-time priority matches orders, how pre-trade risk checks guard the core, how a single sequencer and single-threaded engine give deterministic low-latency matching, and how market-data feeds, journaling/replay, clearing and symbol partitioning fit together.

Low-latencyConcurrencyDeterminism
System Design

Design a CDN

Build a Content Delivery Network like Cloudflare or Akamai. See why distance is the enemy, how edge PoPs cache content near users, how anycast/GeoDNS routes to the nearest edge, how Cache-Control TTLs and tiered shield caches protect the origin, how versioned URLs and purge solve invalidation, and how stale-while-revalidate keeps it fast and up.

CachingEdgeScalability
System Design

Design DNS

Build the Domain Name System. See why one central name→IP table can't work, how a delegated hierarchy of root, TLD and authoritative servers splits the namespace, how a recursive resolver walks it, how caching with TTLs makes lookups instant, how replication and anycast make it DDoS-proof, and how GeoDNS, DNSSEC and propagation work.

NetworkingCachingAvailability
System Design

Design a Feature Flag System

Build a feature flag / experimentation platform like LaunchDarkly. See why redeploying to toggle a feature is slow and risky, how a flag store + dashboard lift decisions out of code, how an SDK evaluates flags locally with zero network calls, how consistent-hash bucketing gives sticky percentage rollouts, how local cache + streaming + edge deliver a millisecond kill switch, and how exposure events turn a flag into an A/B experiment.

ScalabilityCachingReliability
System Design

Design a Leaderboard

Build a real-time leaderboard for millions of players. See why SQL ORDER BY + COUNT rank melts at scale, how a sorted set (Redis ZSET) gives O(log N) update/rank and O(log N + K) top-K, how relative "around me" boards work, why sharding a global ranking is uniquely hard, how time-windowed boards reset cleanly, and how a durable store backs the rebuildable in-memory index.

CachingScalabilityReal-time
System Design

Design a Distributed Counter

Build a distributed counter for view counts and likes at massive scale. See why a single UPDATE row melts under write contention, how sharding into N sub-counters trades read cost for parallel writes, how batching through a stream cuts write volume, how a cached total makes reads O(1), how HyperLogLog counts uniques in a few KB, and how idempotency keys make increments exactly-once.

CachingScalabilitySharding
System Design

Design a Live Streaming System

Build a live video streaming system like Twitch or YouTube Live. See why it optimizes for cheap CDN fan-out instead of ultra-low latency, how ingest accepts one broadcaster push, how transcoding builds an adaptive bitrate ladder, how HLS/DASH segmenting turns live video into cacheable HTTP files, how a CDN fans them to millions, the latency-vs-scale dial, and how live chat and DVR/VOD fit in.

StreamingCachingScalability
System Design

Design a Flash Sale System

Build a flash-sale / limited-inventory system for a massive concurrent drop. See why read-then-decrement oversells, how admission control and a virtual waiting room pace the thundering herd, how an atomic inventory decrement guarantees exactly N buyers win, how reserve-then-confirm with a hold TTL and async order processing work, and how to tame the single hot inventory key with fairness and anti-bot defenses.

ConcurrencyScalabilityCaching
System Design

Design a Hotel Booking System

Build a hotel/room booking platform like Booking.com or Marriott. See why read-then-decrement inventory oversells rooms, how a multi-night stay needs an all-or-nothing atomic transaction across every date, how reserve-then-confirm with a hold TTL prevents both overselling and cart-abandonment loss, why dynamic pricing is cached rather than computed live, how search is split from the strongly-consistent booking path (CQRS), and how controlled overbooking is a business policy the same atomic system enforces.

ConcurrencyConsistencyCaching

AI System Designs 1

AI System Design

Design an LLM Cache

Build a prompt & response caching layer for LLMs. See why repeated and near-duplicate prompts should skip the model, how an exact-match cache keys on the normalized request, how a semantic cache reuses answers above a tuned similarity threshold, how prompt-prefix (KV) reuse cheapens even misses, how TTL and invalidation keep it fresh — and how a loose threshold silently serves wrong answers.

LLMCachingInfrastructure

Coding Challenges 8

Challenge

LRU Cache

The eviction policy behind every size-limited cache: when you run out of room, throw out whatever was used least recently. The trick is O(1) get and put — an ordered hash map (Python dict / JS Map) gives you exactly that. Solve it in Python or TypeScript.

SystemsCachingData Structures
Challenge

Streaming Median

Latency dashboards do this every second: maintain the median of a stream without re-sorting per event. The classic two-heap trick — a max-heap for the low half, a min-heap for the high half, the median always at the boundary. Solve it in Python or TypeScript.

Data StructuresHeapsSystems
Challenge

Token Bucket Rate Limiter

The algorithm inside most production rate limiters — and it never runs a timer. Refill the bucket lazily from the time elapsed since the last request, cap at capacity, spend one token or reject. Two numbers of state per client, exactly like the Redis version. Solve it in Python or TypeScript.

SystemsRate LimitingDistributed Systems
Challenge

Sliding-Window Rate Limiter

Allow at most N requests per rolling window — the rate limiter that guards real APIs. A sliding log of accepted timestamps gives exact limits without fixed-window bursts. Decide accept/reject for a stream. Solve it in Python or TypeScript, with hidden tests.

SystemsRate LimitingQueue
Challenge

LFU Cache

The cache that evicts what you use least often — and, on ties, least recently. Harder than LRU: track frequency and recency together, still O(1) per op. Replay get/put operations. Solve it in Python or TypeScript, with hidden tests.

DesignCachingHash Map
Challenge

Bloom Filter

A tiny bit-array that answers "have I seen this?" in a fraction of a set’s memory — with occasional false positives but never a false negative. Build one with double hashing. Provided hashes keep both languages in sync. Hidden tests.

Data StructuresProbabilisticHashing
Challenge

HyperLogLog Cardinality Estimator

Count how many distinct items a stream held — billions of them — using a few kilobytes, not a giant set. Turn "the longest run of leading zeros" into an estimate. Powers COUNT(DISTINCT) in Redis and BigQuery. Solve it in Python or TypeScript, with hidden tests.

ProbabilisticStreamingHashing
Challenge

KV-Cache Eviction (Attention Sinks)

An LLM’s KV cache grows every token, so long chats must drop old entries without wrecking quality. StreamingLLM keeps the first few "attention sink" tokens plus a sliding window, evicting the middle. Compute the survivors. Solve it in Python or TypeScript, with hidden tests.

AIInferenceCaching

Labs 2

Lab

The Eviction Race: LRU vs LFU

Don't read about cache eviction — race the policies. Feed the same access stream to three caches of equal size and watch them make different choices: FIFO evicts the oldest, LRU evicts the least-recently-used, LFU evicts the least-frequently-used. On a hot-key-plus-scan workload, LFU keeps the hot item while LRU and FIFO throw it away — and the hit rates diverge. Step the stream and watch, made playable, with theory and a quiz.

SystemsCachingPerformance
Lab

Backpressure: The Bounded Queue

Don't read about backpressure — feel the queue fill. A fast producer feeds a slow consumer through a bounded buffer. Without backpressure the queue overflows and you silently drop work; with backpressure you slow the producer to the consumer's pace so nothing is lost — you trade latency for reliability. Toggle backpressure and watch drops turn into controlled waiting, made playable, with theory and a quiz.

SystemsBackpressurePerformance

Interactive Tools 6

Tool

Interactive Capacity Estimator

Slide DAU, requests per user, payload size and read/write ratio and watch QPS, storage per year, bandwidth, shard count and cache memory recompute live — every number with the formula behind it. The napkin math interviewers expect, made interactive.

System DesignCalculatorInterview
Tool

Percentile Calculator

Drop in a list of latencies (or any numbers) and get p50, p90, p95 and p99 instantly, plus min, max and mean — with the sorted distribution drawn as a histogram and each percentile marked on it. See exactly why the average hides your worst requests and the tail is what users feel.

System DesignCalculatorReliability
Tool

Token Bucket Simulator

A live token-bucket rate limiter you can play. Set the bucket capacity and refill rate, then send requests by hand or turn on auto traffic — watch the bucket drain when you burst, refill when you pause, and reject requests once it hits empty. The algorithm behind almost every production rate limiter, made tactile.

Distributed SystemsRate LimitingVisualizer
Tool

Latency Budget Builder

Assemble a chat request stage by stage — network, gateway, retrieval, rerank, prefill, decode — and watch the waterfall: time to first token, full response time, a stacked bar of where the milliseconds went, and which stage to attack first when the budget blows.

AILLMSystem Design
Tool

Cache ROI Calculator

A cache ROI calculator. Enter your request rate, cache hit rate, and the latency of a hit versus a miss to see your effective average latency, how much faster that is than going to the origin every time, and how much backend load the cache removes. Add a per-origin-request cost to estimate the money saved.

CachingCalculator
Tool

Latency Numbers Explorer

An interactive version of the classic "latency numbers every programmer should know." Compare the real time of an L1 cache reference, a memory read, an SSD read, a same-datacenter round trip and a cross-continent round trip — then scale them all to human time (where one CPU cycle is a second) to feel just how enormous the gaps really are.

Low-latencyReference

About caching & performance

Almost every fast system is fast because it avoids work — and caching is how. A cache keeps the results of expensive operations close and cheap to fetch, turning a database query or a network round-trip into a sub-millisecond lookup. The art is in what to cache, how to evict (LRU, LFU, TTL), and how to stay correct as the underlying data changes.

Beyond caches, performance is about the whole path: CDNs that move content to the edge, rate limiters that protect a service from overload, and capacity math that sizes it all. This topic collects the tools and systems where hit rate and tail latency are the difference between a snappy product and a slow one.

More in Systems & Backend

Distributed Systems → Databases & Storage → Real-time & Messaging → Forward Deployed Engineering → Developer Utilities →
← Browse all topics