The whole design, in writing
Learn system design by building a flash-sale / limited-inventory system (think a 1000-unit drop with millions of buyers, or Ticketmaster-style scarcity) step by step. An interactive guide covering why a read-then-decrement oversells, atomic inventory decrement, admission control and a virtual waiting room to survive the thundering herd, reserve-then-confirm with async order processing, fairness and anti-bot, and the unhappy paths (reservation expiry, hot-key, idempotency).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What makes a flash sale hard?
At 12:00:00 sharp, a 1000-unit drop opens and two million people slam "buy" in the same second. Two things must both hold: you sell exactly 1000 units — never a single one more (overselling a sold-out item is a real, costly failure) — and the site stays up under a spike 1000× its normal load. Scarcity plus a thundering herd.
A flash sale system guarantees no oversell under extreme concurrency while surviving the traffic spike. Two independent problems: correctness (an atomic inventory decrement so item 1001 can never be sold) and scale (admission control + a queue so the herd never crushes the core). Reserve fast, confirm async.
What the new pieces do
- Buyersclient
- A massive, synchronized spike — millions hitting "buy" at 12:00:00 for a few thousand units. The thundering herd.
Step 1 · The oversell bug
Read-then-decrement races
The naive purchase: read stock, if it's > 0 decrement it and give the user a unit. Under millions of concurrent buyers, why does this sell more units than exist?
read stock; if > 0, decrement. Why does this oversell under massive concurrency?
Space isn't it. The bug is a race: read and decrement are two steps, so thousands of buyers can all read "5 left" at the same instant and all pass the check, then all decrement — selling far more than 5.
Type isn't the issue. The flaw is that check-then-decrement is non-atomic: many concurrent buyers read the same positive stock before any decrement lands, so they all "win" and oversell.
With a gap between reading stock and decrementing it, massive concurrency means many buyers observe the same "in stock" value before any of them writes, so they all proceed and the count goes negative. You need the check-and-decrement to be one atomic step.
Read-then-decrement is two steps, so under massive concurrency thousands of buyers all read the same "in stock" value before any decrement lands — they all pass the check and all decrement, driving stock negative. That's overselling. The fix is to make "if available, take one" a single atomic operation that the store serializes.
Why this piece earns its place
This bug is dangerous because it passes every test you are likely to write. The window between the read and the write is microseconds wide, so on a laptop, in CI, and on a staging box with ten users, read-then-decrement is indistinguishable from correct code — the race only exists in the one second of the year when the whole herd arrives together, which is also the one second you cannot rehearse. Afterwards the logs defend it: every request returned success, because from each buyer’s point of view the check really did pass. You find out from the sold count. So I wouldn’t treat this as a code-review problem. Make the invariant structural — stock cannot go negative anywhere in the system, enforced by the store itself — so the wrong pattern fails loudly in the paths nobody thought to test, instead of quietly on the day of the drop.
Step 2 · Survive the flood
Admission control & the waiting room
Even with correct inventory, two million requests hitting your core services in one second will melt them. You can't scale to serve everyone instantly for a few thousand units. How do you protect the system from the herd itself?
Two million requests in one second for 1000 units. How do you protect the core?
A front gate absorbs the flood: it rate-limits, filters bots, and lets users through in manageable batches (or a "virtual waiting room" that queues everyone and admits N/sec). The core services only ever see a controlled trickle, not the full spike.
Scaling to serve two million in one second for a thousand units is wildly expensive and often impossible fast enough — and pointless, since almost everyone will fail to get a unit. Shed and pace the load at the edge instead.
The database will not cope with a 1000× spike; it will fall over, failing everyone including the winners. You must throttle admission before traffic reaches the core.
Put admission control in front: a gate (often a virtual waiting room) that rate-limits, filters bots, and admits users in controlled batches — e.g. queue everyone and let N per second through to actually attempt a purchase. The core services only ever see a manageable trickle, so a 1000× spike becomes a steady, survivable stream. Most users wait in line; the system stays up.
Why this piece earns its place
Admitting N per second turns a flood into a rate, and that rate is the knob. The mistake is deriving it from the inventory number — a 1000-unit drop does not imply a thousand admissions. Set it from the measured capacity of the narrowest thing behind the gate, which here is the order service turning wins into reservations, and leave headroom, because the line is what you have instead of an outage. Too high and the trickle is still a flood: the core falls over and the waiting room’s promise becomes a lie. Too low and the sale crawls — people who are genuinely in line give up, and the drop can end with units never claimed. The gate itself has to be the cheapest thing in the system, because everybody touches it and almost nobody gets past it: a static page and a signed token, served at the edge, holding no per-user server state.
- waiting roomqueue the herd
- N/secadmit in batches
- edgeshed load early
What the new pieces do
- Admissiongateway
- The front gate: rate-limits, filters bots, and admits users in controlled batches so the core never sees the full flood.
Step 3 · Never sell 1001
Atomic inventory
Admitted users now attempt to grab a unit. This is the correctness core: how do you let exactly 1000 succeed and everyone after them get "sold out", under heavy concurrency, fast?
Admitted buyers grab units. How do you let exactly 1000 succeed, fast, with no oversell?
Keep remaining stock in an atomic in-memory counter and decrement it atomically per purchase; the buyer wins only if the post-decrement value is still ≥ 0, otherwise it's sold out (and you can undo). Atomicity guarantees exactly the available count succeed — no oversell — at very high throughput.
A table lock serializes every purchase through one lock — correct but far too slow for the throughput, creating a huge queue. An atomic counter (or a conditional decrement) achieves correctness with far higher throughput.
Overselling a scarce item and clawing it back is a terrible, trust-destroying experience (and legally fraught). Prevent it up front with an atomic decrement — never sell the unit that doesn't exist.
Hold remaining stock in a fast atomic counter (Redis DECR / a Lua script, or a DB UPDATE … SET stock=stock-1 WHERE stock>0). Each purchase does one atomic decrement; the buyer wins only if the result stays ≥ 0, otherwise it's sold out (release the decrement). Atomicity guarantees exactly the available number of buyers can ever succeed — item 1001 is never sold — at in-memory throughput.
Why this piece earns its place
An atomic decrement guarantees you never sell one too many. It does not guarantee you sell them all, and the gap is the acknowledgement. If the counter applies the decrement and the reply never reaches the caller — a timeout, a dropped connection, a process restarting between the decrement and writing the reservation down — the unit is gone from stock and no buyer knows they own it. The user sees sold out and retries, which costs another unit, and a scarce drop can bleed real inventory to nothing but network weather. So the decrement isn’t fire-and-forget arithmetic: it returns a claim, recorded under a key derived from the buyer, so a retry from that same buyer re-reads the claim it already holds instead of taking a second one. Everything downstream keys off that claim — and a claim nobody ever picks up is reclaimable, which is what the expiry path exists for.
- DECRatomic, no race
- ≥ 0 winselse sold out
- exactly Ncan ever succeed
What the new pieces do
- Inventorycache
- The fast atomic counter of remaining units. An atomic decrement is the gate that makes overselling impossible.
Step 4 · You won — briefly
Reserve, don't sell yet
A winning decrement means the user gets to buy — but they still have to pay, which takes seconds and can fail. If you only decrement on successful payment, slow buyers block the unit; if you sell before payment, non-payers steal units. What state is the unit in right after the grab?
Model the win as a reservation, not a final sale. A successful decrement reserves the unit for that user with a short hold TTL (e.g. 5–10 minutes) to complete checkout. The unit is neither freely available nor permanently sold — it's held. If they pay, it becomes a confirmed order; if the hold expires, the reservation is cancelled and the unit is returned to stock (an atomic increment) for someone else.
Why this piece earns its place
The hold length is the number I’d argue about. Measure it from how long real checkout takes at its slow tail — a card that triggers an extra verification step, a banking app on a phone on mobile data, someone fetching a wallet from another room — not from how fast you can click through it. Too short and you take the unit back from a buyer who won it fairly and is actively paying, which is the worst outcome in the room because they watch it happen. Too long and the drop reads sold out while units sit dead behind abandoned checkouts. The hold also has to outlive the payment attempt it protects, or the two race: the TTL fires, the unit returns to stock and is won again, and then the original charge succeeds. So I’d size the hold as the payment attempt’s own deadline plus a margin, and have the worker give up on a charge it can no longer land rather than let it succeed against a unit somebody else already owns.
- reservehold with TTL
- confirmpay → order
- expirerelease back to stock
What the new pieces do
- Order Serviceservice
- Turns a successful decrement into a reservation, then an order — quickly handing slow work to the queue.
Step 5 · Decouple the slow parts
Async order processing
Payment, fraud checks, inventory persistence, notifications and fulfillment are all slow relative to the instant decrement. Doing them synchronously while the user (and thousands of admitted others) wait would jam the whole flow. How do you keep the grab fast?
Payment/fulfillment are slow. How do you keep the fast "grab" from being blocked by them?
Blocking the fast reservation on slow payment/fulfillment jams the whole flow under load and couples the quick gate to slow, fallible steps. Hand the slow work to a queue and process it asynchronously.
You can't skip payment or fulfillment. You decouple them: the reservation is instant, and the slow work happens off the hot path via a queue, holding the unit until it completes.
The reservation is instant; the confirmed grab is put on a durable queue, and workers handle payment, persistence and fulfillment at their own pace while the hold TTL protects the unit. The fast path stays fast; slow, fallible work is isolated and retryable.
Once a unit is reserved, enqueue the order on a durable queue and process the slow work — payment, persistence to the durable store, fraud checks, fulfillment, notifications — asynchronously off the hot path. The user gets an instant "you're in, completing your order" while workers handle it at their own pace, protected by the reservation hold. The fast reservation path never waits on slow, fallible steps.
Why this piece earns its place
Handing the slow work to a queue quietly assumes one thing: that the queue drains faster than the hold lasts. While that holds, a backlog is just latency and the reservation covers it. The day it stops — workers crashlooping, the payment provider degraded, consumers falling behind producers — reservations start expiring while their own orders are still in the queue waiting to be paid for. A worker then picks up an order whose unit is already back on sale, and if it charges anyway the same unit has been sold twice. So consumer lag isn’t an ops metric here, it’s a correctness signal: compare it against the hold budget continuously, have the worker re-check that the reservation is still live immediately before it charges rather than trusting the message, and throttle admission at the gate when lag approaches the hold. Better to slow the line than to sell units the system has already given away.
- enqueueconfirmed reservation
- asyncpayment + fulfillment
- retryabledurable queue + workers
What the new pieces do
- Order Queuebus
- Buffers confirmed reservations so payment and fulfillment happen asynchronously, not on the hot path.
- Durable Storedata
- The durable record of orders and the ultimate inventory truth — reconciled with the fast counter, never oversold.
Step 6 · Money & completion
Payment and confirmation
The async worker now has to actually charge the winner and turn the reservation into a real order — and payments are slow, can fail, and must never double-charge.
The worker takes the reserved order and runs payment with an idempotency key (so a retry never double-charges), then, on success, confirms the order in the durable store — the reservation becomes a paid, fulfilled order. On payment failure or timeout, cancel the reservation and release the unit back to stock (atomic increment) so it isn't lost. Notify the user of the outcome. Payment is just a reliable, idempotent step in the async pipeline, with the hold TTL as its deadline.
Why this piece earns its place
An idempotency key only works if two attempts at the same purchase produce the same key. Minted fresh per attempt — a new identifier generated when the worker retries — it’s decoration, and the second attempt charges again. It has to derive from something stable that both attempts share, and the reservation is exactly that: one reservation, one key, one charge, however many times the queue redelivers it. The release path needs the same suspicion, because the increment that returns a unit to stock is atomic but not idempotent. Run it twice for one reservation — an expiry sweep and a failed-payment handler both deciding this unit is free — and stock is now larger than the drop, which is overselling arriving through the back door after all the work spent shutting the front. So the reservation row is the only place a release is allowed to be recorded. The expiry sweep and the failed-payment handler both compare-and-set held → released on that row, and only whoever wins that write runs the increment; the loser reads a row that is already released and does nothing at all, quietly, exactly as if it had never fired.
- idempotency keyno double-charge
- confirmreservation → order
- fail → releaseunit back to stock
What the new pieces do
- Paymentservice
- Charges the winner. On success the reservation becomes a paid order; on timeout the held unit is released.
Step 7 · The hot single item
The hot-key & fairness
Everyone is contending on one inventory counter — a single, blazing-hot key that every purchase hits, plus determined bots trying to grab everything and the need for the sale to feel fair.
Tame the hot key: the atomic counter is one cell every buyer touches, so keep it in a fast in-memory store, optionally shard the stock into buckets (1000 = 10×100) so decrements spread across keys, and pre-warm/replicate it. Enforce fairness & anti-abuse: per-user purchase limits, rate limiting and bot filtering at admission, CAPTCHAs, and pre-issued queue tokens so one script can't sweep the drop. Persist the fast counter's state to the durable store and reconcile, so the in-memory gate and the system of record never disagree — and the durable store is the final backstop against oversell.
Why this piece earns its place
Sharding the counter buys throughput and sells you a tail. Split the stock into buckets and remaining stock is no longer one number anyone can read — a buyer routed to a bucket that happens to be empty is told sold out while units sit in the bucket next door. So a miss can’t be an answer: it becomes a walk across the remaining buckets, and sold out only means anything once every one of them is drained. That makes the last units far more expensive to sell than the first, exactly when the herd is loudest, and it turns bucket sizing into a fairness question, because an unlucky assignment is the difference between winning and not. Which is why I’d shard only when one key is measurably saturating, and start from the smallest number of buckets that clears it. Every extra bucket is more scanning at the tail and more chance a buyer with a unit in front of them is turned away.
- shard stockspread the hot key
- per-user limitfairness + anti-bot
- reconcilecounter ↔ durable truth
Step 8 · The sharp edges
Expiry, idempotency & under-sell
Real drops bring edge cases: units held by users who never pay, duplicate requests from retries and double-clicks, and the opposite risk of under-selling (units stuck in expired holds while the item shows "sold out").
Reclaim abandoned units by expiring reservations and returning them to stock, and re-open the sale if reclaimed units appear — so you don't under-sell (show sold-out while units are stuck in dead holds). Make every step idempotent (idempotency keys, dedupe) so retries, double-clicks and at-least-once queue delivery don't double-reserve or double-charge. Give clear real-time status (in line / you won / sold out) to cut refresh-hammering. And treat the durable store as the final arbiter of inventory, reconciling the fast counter against it so neither oversell nor lost units can persist.
Why this piece earns its place
Reconcile against the durable store is the sentence everyone says, and it’s the one operation that can undo everything above it. A decrement is safe under concurrency because it’s relative. A reconciliation writes an absolute number computed from a snapshot, and during a drop that snapshot is stale before it lands — units claimed while it was being taken get handed back out and the counter goes up. That is the single path left in this design that can oversell after the atomic gate did its job. So reconciliation detects and alarms; it doesn’t correct. Units return to stock only through the same guarded increment a release uses, one per reservation. And when reclaimed units do reopen the sale, the reopen goes back through admission rather than straight to the buyers refreshing a sold-out page — otherwise the small second drop arrives with the same herd as the first and none of the pacing.
You did it
You just designed a flash sale system.
Everything you assembled, in order
- Read-then-decrement oversells under concurrency — thousands read the same stock before any write.
- Admission control / a virtual waiting room paces the herd so the core sees a survivable trickle.
- An atomic decrement (Redis DECR / conditional UPDATE) guarantees exactly N buyers win — never item 1001.
- A win is a reservation with a hold TTL, not a final sale — pay to confirm, or it's released.
- Enqueue confirmed reservations and do payment/fulfillment async off the hot path.
- Payment is idempotent (no double-charge) and reversible (failure returns the unit to stock).
- Tame the single hot inventory key (shard it), enforce fairness/anti-bot, and reconcile to the durable store.