The big idea
What makes payments hard?
Moving money looks like a simple write, but it’s the opposite: it touches external banks you don’t control, must never double-charge or lose a cent, has to survive every retry and crash, and must prove afterward that the books balance to the penny.
Treat a payment as a state machine recorded in an immutable double-entry ledger, made safe against retries by idempotency, run asynchronously around slow processors, and reconciled against the bank’s own records. Correctness first, everywhere.
How to read this: We add one piece at a time, problem then fix, and the diagram grows. Hit Begin.
Step 1 · The flow
Authorize, then capture
“Charge this card $50” isn’t one step. The money has to be checked and held, then actually taken — and the request leaves your system entirely to reach the card networks, which can be slow or fail.
Design decision: "Charge this card $50" — but the request leaves your system to reach slow card networks. How do you model it?
The call: A state machine: authorize (hold) then capture (take), via the PSP. — Splitting auth from capture lets you reserve funds now and finalize later, and explicit states (created→authorized→captured→settled) let you reason about partial failures and retries.
The Payment API hands the charge to a Payment Worker that drives it through states: authorize (hold funds) then capture (take them), talking to the PSP / card networks. Splitting auth from capture lets you reserve now and finalize later.
A payment is a state machine: created → authorized → captured → settled (or failed/refunded). Modeling explicit states — not a boolean “paid?” — is what lets you reason about partial failures and retries correctly.
Step 2 · Never charge twice
Idempotency
Networks time out. The merchant retries the charge, unsure if the first one went through. Naively, that’s two charges for one purchase — the cardinal sin of payments.
Design decision: A charge times out and the merchant retries, unsure if the first went through. How do you avoid double-charging?
The call: Require an idempotency key per charge; replay returns the stored result. — The first request does the work and stores the result under the key; any retry with the same key returns that same result without charging again. Make repeating a request unable to repeat its effect.
Require an idempotency key per logical charge. The first request does the work and stores the result under that key; any retry with the same key returns the same result without charging again. Safe to retry, always.
Make retries free: Because failures are ambiguous (did it succeed before the timeout?), the only safe design is one where repeating a request can’t repeat its effect. Idempotency keys turn “maybe” into “exactly once”.
Step 3 · Track every cent
The double-entry ledger
A single “balance” field that you increment is impossible to audit and easy to corrupt under concurrency. When money goes missing, you need to know exactly where — and prove it.
Design decision: How do you record balances so money is auditable and can’t be silently corrupted under concurrency?
The call: An immutable double-entry ledger; balances are derived sums. — Every transaction debits one account and credits another by equal amounts, append-only, never edited. Debits must equal credits so the books always sum to zero — a continuous built-in correctness check.
Record every movement in an immutable, double-entry Ledger: each transaction debits one account and credits another by equal amounts, and rows are never edited — only new entries appended. Balances are derived by summing entries.
Debits equal credits: Double-entry has kept money honest for 500 years. Every cent leaving one account arrives in another, so the books must always sum to zero — a built-in, continuous correctness check.
Step 4 · Don’t hold the cards
Tokenization & PCI
Storing raw card numbers makes you a giant target and drags your entire system into the strictest PCI compliance scope. One leak is catastrophic — and most of your services never need the real number anyway.
Design decision: You need to charge cards repeatedly but storing card numbers makes you a giant breach target. What do you do?
The call: Send cards to a PCI vault that returns a token; store only tokens. — Card data goes straight into a hardened vault that returns a token; your API, workers and ledger handle only meaningless tokens. Raw PANs live solely in the vault and at the network — shrinking PCI scope and blast radius.
Send card data straight into a PCI-compliant Token Vault, which returns a token. Everything else — your API, workers, ledger — stores and passes only tokens; raw card numbers exist solely inside the vault and at the network.
Shrink the blast radius: Tokenization confines sensitive data to one hardened component. The rest of the system handles meaningless tokens, slashing both PCI scope and the damage any breach could do.
Step 5 · Slow banks, fast API
Async events & webhooks
Authorization and settlement can take seconds — or arrive minutes later (a bank confirmation, a delayed failure). Blocking the merchant’s request until everything finishes is both slow and impossible.
Design decision: Authorization and settlement can take seconds or arrive minutes later. How does the API respond fast?
The call: Emit state changes to an event stream; webhook the final outcome. — The API returns the current state immediately; a durable event stream drives webhooks that notify the merchant of the final result (with retries) once it arrives. Slow external steps run off the request path.
Emit each state change to a durable Payment Events stream. The API responds quickly with the current state; Webhooks later notify the merchant of the final outcome (with retries). Slow, external steps run asynchronously off the request path.
Decouple from the slow world: You don’t control the banks’ latency, so don’t couple your response to it. Events + webhooks turn an inherently asynchronous, multi-party process into something each side can handle at its own pace.
Step 6 · Prove it balances
Reconciliation
Your ledger says one thing; the processor’s settlement may say another, because of fees, declines, chargebacks, or a missed event. Over millions of payments, tiny discrepancies hide real lost money.
Design decision: Your ledger and the processor’s settlement can disagree (fees, declines, chargebacks, a missed event). How do you catch lost money?
The call: Reconcile the ledger against PSP statements daily, flag mismatches. — A daily job compares your ledger line-by-line against the bank’s settlement statements — external truth — and flags every mismatch for investigation. It catches whatever the live path missed.
A daily Reconciliation job compares your Ledger against the PSP’s settlement statements, line by line, and flags every mismatch for investigation. The bank’s record is treated as external truth your internal books must agree with.
Trust, then verify: Distributed systems drift; money systems can’t. Reconciliation is the safety net that catches whatever the live path missed, turning “we think it’s right” into “we’ve proven it’s right”.
Step 7 · Consistency & scale
The sharp edges
A capture succeeds at the PSP but your ledger write fails — now your records and the bank disagree. And at scale, hot merchants and refunds/chargebacks add states and load that strain the system.
Make the worker a durable, retryable state machine (outbox pattern: persist intent, then act, then mark done) so it always converges even after crashes. Shard by merchant/account, and model refunds and chargebacks as first-class ledger transactions, never edits.
Converge, don’t assume: Across systems you can’t get atomicity, so design for eventual consistency with a relentless reconciler: record intent durably, retry until the external side agrees, and let the ledger be the immutable history of it all.
You did it
You just designed a payment system.
- A payment modeled as a state machine: authorize then capture via the PSP.
- Idempotency keys make every charge safe to retry — exactly one charge per key.
- An immutable double-entry ledger keeps every cent auditable and balanced.
- Tokenization confines card data to a PCI vault, shrinking the blast radius.
- Async events + webhooks decouple the fast API from slow, external banks.
- Daily reconciliation against PSP statements proves the books actually agree.
- A durable, retryable worker plus refunds-as-ledger-entries handle failure and scale.