Latency vs Throughput: The Two 'Faster's People Confuse
Latency is how long one request takes end to end — what a single user feels. Throughput is how many requests the system handles per second — the fleet's capacity. Below capacity they barely interact; push toward the limit and queues form, and latency spikes even though nothing about a single request changed.
Latency is how long one request takes end to end — what a user feels, best measured at p99, not the average. Throughput is how many requests the system handles per second — its total capacity. Below capacity they barely touch: adding servers buys more throughput without making any single trip faster. Push load toward the limit, though, and queues form, latency spikes, and the two collide — the hockey-stick curve every system eventually hits. Little's Law ties them together: requests in flight equals throughput times latency.
Transcript
Your app feels too slow, so you spend a month making it faster — and users swear it's exactly as slow as before. How is that even possible? Because 'faster' means two opposite things — latency and throughput — and confusing them wastes months. So picture a highway between two cities.
Latency is how long ONE trip takes — your car, on-ramp to off-ramp. For a server, how long a single request takes end to end. It's what a user actually FEELS: the click-to-response wait, measured in milliseconds. And you watch the worst cases — the p99, not just the average.
Throughput is how many cars pass a point per minute — how many requests the system completes per second. It's about the whole fleet, not your one car. It's what capacity planning lives on: requests per second, the load the highway can carry before it jams.
Here's the trap. With capacity to spare, they barely interact — add lanes, and far more cars per minute get through, yet your own trip takes exactly as long. Spare capacity buys throughput, not speed. But fill that highway toward its limit and they lock together: cars pile up, queues form, and every trip slows.
So it's a curve, not a constant. Below capacity, latency stays flat while throughput climbs; push to the limit and latency spikes — the hockey stick every system has. And you can trade on purpose: batching — a shuttle waiting to fill every seat — buys throughput by spending each rider's wait. Little's Law: requests in flight equal throughput times latency.
So that lost month? You raised throughput when users were feeling latency — the right fix for the wrong number. 'Make it faster' is a question, not a goal. So next time someone says 'make it faster,' ask: the wait one user feels, or the load the whole fleet carries? One highway, two very different numbers.