The whole design, in writing
Learn system design by building a payment system like Stripe step by step. An interactive guide covering the charge flow, idempotency to prevent double-charges, a double-entry ledger, card tokenization, async processing with webhooks, reconciliation against PSP statements, and consistency at scale.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What makes payments hard?
Moving money looks like a simple write, but it’s the opposite: it touches external banks you don’t control, must never double-charge or lose a cent, has to survive every retry and crash, and must prove afterward that the books balance to the penny.
Treat a payment as a state machine recorded in an immutable double-entry ledger, made safe against retries by idempotency, run asynchronously around slow processors, and reconciled against the bank’s own records. Correctness first, everywhere.
What the new pieces do
- Merchant Appclient
- Wants to charge a customer and get a clear yes/no, without ever touching raw card data or risking a double-charge on a retry.
Step 1 · The flow
Authorize, then capture
“Charge this card $50” isn’t one step. The money has to be checked and held, then actually taken — and the request leaves your system entirely to reach the card networks, which can be slow or fail.
"Charge this card $50" — but the request leaves your system to reach slow card networks. How do you model it?
A boolean call can’t express "funds held but not yet taken," can’t handle the networks failing mid-flight, and gives you nowhere to retry from. Money needs explicit states.
Assuming success loses money when the network declines or fails, and gives the merchant no real answer. You must track the charge to a confirmed outcome.
Splitting auth from capture lets you reserve funds now and finalize later, and explicit states (created→authorized→captured→settled) let you reason about partial failures and retries.
The Payment API hands the charge to a Payment Worker that drives it through states: authorize (hold funds) then capture (take them), talking to the PSP / card networks. Splitting auth from capture lets you reserve now and finalize later.
What the new pieces do
- Payment APIbackend
- The entry point for charges and refunds. Validates the request, enforces idempotency, and orchestrates the payment without exposing card networks.
- Payment Workerservice
- Drives the charge through its states — authorize, then capture — talking to the card networks and recording every transition to the ledger.
- PSP / Networksservice
- The external processors and card networks that actually move money. Slow, occasionally flaky, and entirely outside your control — so isolate them.
Step 2 · Never charge twice
Idempotency
Networks time out. The merchant retries the charge, unsure if the first one went through. Naively, that’s two charges for one purchase — the cardinal sin of payments.
A charge times out and the merchant retries, unsure if the first went through. How do you avoid double-charging?
A fuzzy "looks like a duplicate" check is racy and wrong — two legitimate $50 charges look identical, and a slow first charge may not be visible yet. You need an explicit key, not a guess.
The first request does the work and stores the result under the key; any retry with the same key returns that same result without charging again. Make repeating a request unable to repeat its effect.
A timeout is ambiguous — the charge may have succeeded — so "never retry" means merchants miss real charges or hang waiting. Retrying must be safe, which is the system’s job to guarantee.
Require an idempotency key per logical charge. The first request does the work and stores the result under that key; any retry with the same key returns the same result without charging again. Safe to retry, always.
- 1charge per key
- safeto retry
- storedresult
Back of the envelope
- key per logical charge
- 1 charge per key, no matter how many retries
- first request ⇒ do + store result
- replays return the stored result
- ambiguous timeout ⇒ safe retry
- repeating a request can’t repeat its effect
Step 3 · Track every cent
The double-entry ledger
A single “balance” field that you increment is impossible to audit and easy to corrupt under concurrency. When money goes missing, you need to know exactly where — and prove it.
How do you record balances so money is auditable and can’t be silently corrupted under concurrency?
A mutable balance is impossible to audit (no history of why it changed) and easy to corrupt under concurrent updates. When money goes missing you can’t prove where it went.
A side log that drifts from the authoritative balance gives you two disagreeing sources of truth. The record of movements must BE the source of balances, not a copy of it.
Every transaction debits one account and credits another by equal amounts, append-only, never edited. Debits must equal credits so the books always sum to zero — a continuous built-in correctness check.
Record every movement in an immutable, double-entry Ledger: each transaction debits one account and credits another by equal amounts, and rows are never edited — only new entries appended. Balances are derived by summing entries.
What the new pieces do
- Ledgerstore
- The immutable, double-entry record of every movement of money. The single source of truth for balances — append-only, never edited.
Back of the envelope
- every txn: debit = credit
- the books always sum to zero
- append-only, never edited
- a full auditable history of every cent
- balance = Σ entries
- derived, not a mutable field
Step 4 · Don’t hold the cards
Tokenization & PCI
Storing raw card numbers makes you a giant target and drags your entire system into the strictest PCI compliance scope. One leak is catastrophic — and most of your services never need the real number anyway.
You need to charge cards repeatedly but storing card numbers makes you a giant breach target. What do you do?
Even encrypted, raw card data in your database drags every service into the strictest PCI scope and is catastrophic if leaked. Most of your system never needs the real number.
Card data goes straight into a hardened vault that returns a token; your API, workers and ledger handle only meaningless tokens. Raw PANs live solely in the vault and at the network — shrinking PCI scope and blast radius.
Pushing card storage onto every merchant multiplies breach targets and PCI burdens, and still sends raw cards across your system. Centralize sensitive data in one vault, not everywhere.
Send card data straight into a PCI-compliant Token Vault, which returns a token. Everything else — your API, workers, ledger — stores and passes only tokens; raw card numbers exist solely inside the vault and at the network.
What the new pieces do
- Token Vaultstore
- Stores card details in a PCI-compliant vault and returns a token. The rest of the system handles only tokens, never raw PANs — shrinking PCI scope.
Step 5 · Slow banks, fast API
Async events & webhooks
Authorization and settlement can take seconds — or arrive minutes later (a bank confirmation, a delayed failure). Blocking the merchant’s request until everything finishes is both slow and impossible.
Authorization and settlement can take seconds or arrive minutes later. How does the API respond fast?
Settlement can take minutes or arrive later as a delayed confirmation — you can’t hold an HTTP request that long, and you don’t control bank latency. Blocking couples your response to the slowest external party.
Polling wastes requests and still misses results that arrive much later (delayed failures, bank confirmations). Push the outcome when it happens, don’t have the merchant keep asking.
The API returns the current state immediately; a durable event stream drives webhooks that notify the merchant of the final result (with retries) once it arrives. Slow external steps run off the request path.
Emit each state change to a durable Payment Events stream. The API responds quickly with the current state; Webhooks later notify the merchant of the final outcome (with retries). Slow, external steps run asynchronously off the request path.
What the new pieces do
- Webhooksservice
- Pushes final payment status (succeeded/failed/refunded) to the merchant asynchronously, with retries, since results often arrive after the initial response.
- Payment Eventsbus
- A durable stream of payment state transitions that drives webhooks, ledger updates, analytics and reconciliation — decoupling slow steps from the API.
Step 6 · Prove it balances
Reconciliation
Your ledger says one thing; the processor’s settlement may say another, because of fees, declines, chargebacks, or a missed event. Over millions of payments, tiny discrepancies hide real lost money.
Your ledger and the processor’s settlement can disagree (fees, declines, chargebacks, a missed event). How do you catch lost money?
The live path can miss events, and fees/declines/chargebacks alter what actually settled. Over millions of payments, tiny undetected discrepancies hide real lost money. "We think it’s right" isn’t enough for money.
A daily job compares your ledger line-by-line against the bank’s settlement statements — external truth — and flags every mismatch for investigation. It catches whatever the live path missed.
Silently overwriting your ledger destroys the audit trail and hides the bug causing the drift. Mismatches must be flagged and investigated, corrections recorded as new entries — never edits.
A daily Reconciliation job compares your Ledger against the PSP’s settlement statements, line by line, and flags every mismatch for investigation. The bank’s record is treated as external truth your internal books must agree with.
What the new pieces do
- Reconciliationworker
- Compares your ledger against the PSP’s settlement statements daily, flagging any mismatch — the audit that proves the books actually balance.
- PSP Statementsstore
- The processor’s record of what truly settled and when. Reconciliation treats this as external truth to verify your internal ledger against.
Back of the envelope
- ledger vs PSP statement, daily
- line-by-line external verification
- flag every mismatch
- fees, declines, chargebacks, missed events
- external truth, then investigate
- proven correct, not assumed
Step 7 · Consistency & scale
The sharp edges
A capture succeeds at the PSP but your ledger write fails — now your records and the bank disagree. And at scale, hot merchants and refunds/chargebacks add states and load that strain the system.
Make the worker a durable, retryable state machine (outbox pattern: persist intent, then act, then mark done) so it always converges even after crashes. Shard by merchant/account, and model refunds and chargebacks as first-class ledger transactions, never edits.
You did it
You just designed a payment system.
Everything you assembled, in order
- A payment modeled as a state machine: authorize then capture via the PSP.
- Idempotency keys make every charge safe to retry — exactly one charge per key.
- An immutable double-entry ledger keeps every cent auditable and balanced.
- Tokenization confines card data to a PCI vault, shrinking the blast radius.
- Async events + webhooks decouple the fast API from slow, external banks.
- Daily reconciliation against PSP statements proves the books actually agree.
- A durable, retryable worker plus refunds-as-ledger-entries handle failure and scale.