The whole design, in writing
Learn system design by building a digital wallet like PayPal, Venmo or a bank balance system step by step. An interactive guide covering why a mutable balance column is dangerous, the append-only double-entry ledger where balance is derived, atomic debit+credit transfers with idempotency, preventing double-spend under concurrency, sharding accounts with cross-shard transfers, pending vs settled states, reconciliation, and the unhappy paths.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What makes a wallet hard?
Moving a number from one account to another sounds like a -= 10; b += 10. But this is money: it must never be created, lost, or spent twice — even when requests retry, servers crash mid-transfer, two payments race for the same balance, or a bank takes three days to confirm. Correctness isn't a feature here; it's the whole product.
A digital wallet holds balances and moves money with absolute correctness and a complete audit trail. The core is not a balance you edit — it's an append-only double-entry ledger where balance is derived, every transfer is an atomic debit+credit, and every action is idempotent and auditable.
What the new pieces do
- Userclient
- Someone sending money, topping up, or checking their balance. Expects their money to be exactly right, always.
Step 1 · The dangerous shortcut
A mutable balance column
The naive model: an accounts table with a balance column you UPDATE. A transfer is "subtract from A, add to B." Name everything that can go wrong.
A balance column you UPDATE for each transfer. What's wrong for *money*?
The math being right per-operation isn't enough. Concurrent transfers race (lost updates), a crash between the two updates loses or creates money, and there's no history to audit or reverse. Money needs atomicity, isolation, and an immutable trail.
Overdraft is one problem, but the deeper ones are concurrency (two transfers reading the same balance and both succeeding), partial failure (crash between debit and credit), and no auditable history. A mutable column has no answer for any of these.
Two concurrent transfers can both read balance 100 and both subtract, losing money; a crash after the debit but before the credit destroys money; and overwriting the balance leaves no record of how it got there. Money demands atomic, isolated, append-only accounting.
A mutable balance is unsafe for money on three fronts: concurrency (two transfers both read the old balance and one update is lost), partial failure (a crash after debiting A but before crediting B creates or destroys money), and no audit trail (overwriting a number erases how it got there — you can't reconcile or reverse). Money needs atomicity, isolation, and an immutable history.
Why this piece earns its place
The first instinct in the room is that locking saves the column — wrap the two updates in a transaction, take a SELECT … FOR UPDATE on both rows, and the race and the half-transfer both go away. That is true, and it is still the wrong design, because it fixes one of the three problems and cannot touch the other two. Once the update commits, the previous balance is gone. When a customer disputes a charge, or a bad deploy ran all week, nothing records which movements produced the number, so you can’t explain it, can’t attribute it, and can’t reverse only the affected transfers — the one repair available is to write another number on top, which is the same destructive act that caused the trouble. A ledger gives you the locking as well; the column can never give you the history. That is why the ledger is the base of the design rather than a reporting feature bolted on beside it.
Step 2 · Record movements, don't overwrite
The wallet service + an append-only log
If overwriting a balance is the sin, what do you store instead? You still need to know each account's balance, but without ever mutating a number in place.
If you never overwrite a balance, how do you know what it is?
Record each debit and credit as an append-only entry that's never changed. An account's balance is the sum of its entries — derived, not stored-and-mutated — so you have a complete, auditable history and no destructive updates.
Backups of a mutable column are clumsy, racy, and still lose the per-movement history and atomicity you need. The clean model is an immutable log of movements from which balance is computed.
A side audit log that can drift from the real balance gives you two sources of truth that disagree. Make the log THE source of truth and derive balance from it — one truth, fully auditable.
Introduce a wallet service over an append-only log of money movements. You never edit a balance; you append an immutable entry for each movement. An account's balance is derived as the sum of its entries. Now you have a complete, tamper-evident history, no destructive updates, and a single source of truth — the log — from which everything else (balances, statements) is computed.
Why this piece earns its place
Append-only is a property you enforce, not a habit you rely on. The table still accepts UPDATE and DELETE like any other, so the invariant survives exactly as long as nobody needs it broken — and somebody will, at two in the morning, when a customer is owed money and a one-line update looks faster than thinking. The moment that happens the log stops being evidence: every balance derived from it is unverifiable, and nobody can tell a legitimate history from a patched one. So make it structural. The service’s database role gets INSERT and SELECT on the entries table and nothing else, corrections run through a documented path that appends a reversing entry carrying a reason and an operator id, and the schema treats an entry as closed the instant it is written. Decide now what an entry must carry, too — counterparty, reason, external reference, timestamp — because you cannot backfill a column onto rows you have promised never to edit.
What the new pieces do
- Wallet Servicegateway
- Accepts transfers and balance reads. Validates, dedupes, and delegates the money movement to an atomic transaction.
Step 3 · The accounting invariant
Double-entry, atomically
A transfer moves money between two accounts. How do you record it so that money is provably conserved — never created or destroyed — even if the system crashes halfway through?
How do you record a transfer so money is provably conserved through a crash?
Two independent appends can partially fail — credit B commits, debit A doesn't — creating money. The debit and credit must be one atomic unit that commits together or not at all.
A single-sided entry doesn't enforce that every credit has a matching debit, so nothing guarantees conservation or lets you verify the books balance. Double-entry pairs them by construction.
Double-entry: every transfer is a debit of A and an equal credit of B, written together in a single ACID transaction so both commit or neither. The entries always sum to zero, so total money is conserved by construction, and a crash leaves no half-transfer.
Use double-entry accounting inside an atomic transaction. Every transfer writes a debit of account A and an equal credit of account B as one unit — an ACID transaction so both commit or neither does. The two entries sum to zero, so total money is conserved by construction. A crash between them is impossible: the transaction either fully lands or fully rolls back.
Why this piece earns its place
Notice which accounts this design will eventually fight over. Appending is cheap and stays cheap: two transfers touching the same account just insert two more entries, and inserts do not queue behind each other. What stops being cheap is the moment an account’s spendable balance has to be maintained and checked before money leaves it — a marketplace payout account, a fee account, the platform’s own float. Once that account sits on one side of every transfer in the system, every one of those transfers has to serialize on that account, and the queue grows with traffic no matter how much hardware you buy. The defense is to stop treating it as one account. Split it into sub-accounts — per region, per merchant, per day, or a fixed fan-out chosen by hashing the counterparty — each a real account with its own entries, and define the logical balance as their sum. Double-entry still holds, since every transfer is still a balanced pair; you have only spread the contention across accounts that can be worked independently. Do it before the launch that needs it, because splitting a hot account later means moving money while it is being spent.
- debit + creditone atomic pair
- Σ entries = 0money conserved
- all-or-nothingno half-transfer
What the new pieces do
- Transfer Txnservice
- Runs the debit and credit as one atomic unit — both commit or neither — so a transfer can never half-happen.
- Ledgerdata
- The immutable double-entry record of every movement. Balance is derived from it; entries are never edited or deleted.
Step 4 · Retries that don't double-charge
Idempotency
Networks fail after a request succeeds but before the response arrives, so clients retry. If a retry re-runs the transfer, you've charged someone twice. How do you make "send $10" safe to retry?
A client retries a transfer whose response was lost. How do you avoid charging twice?
The client sends a unique idempotency key with the request; the server records it with the result. A retry with the same key returns the original result without re-applying the transfer — exactly-once, even across retries and crashes.
Pushing correctness onto the user is unreliable and terrible UX, and races still happen. The server must guarantee exactly-once itself via an idempotency key.
Double-charging money is never acceptable. Idempotency keys make retries provably safe, which is mandatory for a payments system.
Require an idempotency key on every transfer. The server records the key with its result in the same transaction as the ledger entries; a retry carrying the same key returns the original outcome without re-applying the transfer. So "send $10" can be retried any number of times and still moves $10 exactly once — even if a crash struck at the worst moment.
Why this piece earns its place
Three details decide whether the key actually holds. First, the dedupe record has to be a uniqueness constraint rather than a read-then-write check: two copies of the same retry can reach two servers in the same millisecond, both look the key up, both find nothing, and both proceed. Let the insert be the thing that decides — the loser takes a constraint violation and returns the winner’s result once it commits. Second, the key identifies a request, not just a caller, so the same key replayed with a different body is a client bug and must be rejected outright; returning the stored result there is worse than an error, because the caller is told their new transfer succeeded when nothing moved. Third, you can’t keep these records forever, and the moment you set a retention window you have created a correctness window — a retry arriving after its key is swept looks brand new and applies again. Keep keys longer than any client will plausibly retry, and bound the sweep by age, never by table size.
- idempotency keyper transfer
- at most onceper key
- retry-safeexactly-once effect
What the new pieces do
- Idempotencycache
- Remembers each request's idempotency key so a retried transfer is applied exactly once, never twice.
Step 5 · Fast reads & no overspend
Materialized balances
Summing an account's entire ledger history on every balance read (and every "do they have enough?" check) gets expensive as history grows. But you also must not let two concurrent transfers both spend the same funds. How?
Summing all ledger entries per read is slow, and you must block overspend. How?
A cache that can drift from the ledger risks approving spend against a wrong balance. The materialized balance must be updated in the same transaction as the ledger so it can never disagree.
Letting money go negative and reconciling afterward is unacceptable for spend authorization. The balance check and the debit must be atomic so two transfers can't both spend the same funds.
Keep a per-account balance updated in the same atomic transaction as the entries, so reads are O(1) and always consistent with the ledger. Guard overspend by checking-and-debiting under a per-account lock (or serialized single-writer), so concurrent transfers can't both pass the check.
Keep a materialized balance per account, updated in the same atomic transaction as the ledger entries — so reads are O(1) and can never disagree with the log (and it's rebuildable from the ledger if lost). Prevent overspend by making the check-and-debit atomic: hold a per-account lock (or serialize each account through a single writer) so two concurrent transfers can't both read "enough funds" and both spend it.
Why this piece earns its place
Two accounts means two locks, and the classic way this fails is taking them in whatever order the request happened to name them. A transfer from A to B and a simultaneous transfer from B to A each grab their sender’s row first and then wait on the other, so the database breaks the tie by killing one — and under load you get a stream of deadlock aborts that look random while clustering on exactly the pairs that trade most. The fix costs nothing: sort the two account ids and always acquire in that order, so the two transfers queue instead of colliding. The second rule is about what you do while holding the lock. Its duration is that account’s serialization window, so anything slow inside it — a fraud check, a call to a partner, an external rail — multiplies directly into how few transfers the account can do per second. Do the slow work before opening the transaction, and let the transaction hold only the read, the check, the two entries and the snapshot update.
- O(1)balance read
- same txnsnapshot = ledger
- per-account lockno double-spend
What the new pieces do
- Balance Snapshotcache
- A materialized per-account balance for fast reads and overspend checks — rebuildable from the ledger.
Step 6 · Beyond one database
Sharding & cross-account transfers
Millions of accounts and their histories outgrow one database. But sharding accounts across databases means a transfer between two accounts can span two shards — and you can't use a single local transaction across them. How do you keep transfers atomic?
Shard by account id so each account's ledger lives on one shard (an account's own transactions stay a local ACID transaction). For a cross-shard transfer, you can't do one local transaction across both, so use a saga (or two-phase commit): model the transfer as a sequence of steps with compensating actions, often via a pending/holds mechanism — reserve funds on A's shard, credit B's shard, then confirm, and compensate (release the hold) if any step fails. Same-shard transfers stay simple and fast; cross-shard ones pay for coordination.
Why this piece earns its place
The part of a saga people underrate is that it has no isolation. Between the reserve on A and the confirm, money sits in a state nothing else understands unless you make it explicit: A’s spendable balance has to be its balance minus its open holds, or the sender simply spends the reserved funds again while they are in flight, and B must not treat the credit as usable until it is confirmed. Then there is the orchestrator. If it dies after crediting B and before confirming A’s hold, nothing on either shard is corrupt and nothing will fix itself, so the saga’s own state is a durable record with a sweeper behind it, and every step carries a key that lets the sweeper re-drive it safely. Be careful what that sweeper does with an old hold. Releasing it on a timer is the tempting rule and it creates money, because the hold may be old precisely because the credit on the other shard already landed. An expired hold is an investigation — resolve it by asking the other shard what happened, never by assuming.
- shard by accountlocal txn per account
- cross-shardsaga / 2PC
- holdsreserve → confirm → compensate
Step 7 · The outside world is slow
Pending vs settled & reconciliation
Top-ups and withdrawals touch banks and card networks that are asynchronous and can take days — and might silently disagree with your books. How do you model in-flight external money, and trust your ledger?
Model money in two states: a pending (authorized/held) entry when an external transfer is initiated, promoted to settled when the bank confirms (or reversed on failure) — so the ledger reflects reality at each stage without pretending an unconfirmed transfer is final. And run continuous reconciliation: match your internal ledger against external bank/rail statements to catch any discrepancy (a settlement that never arrived, a duplicate) and flag it for repair. Trust, but verify against the source of funds.
Why this piece earns its place
Reconciliation only works if you made it possible at write time. Matching a statement line to a ledger entry on amount and date is guesswork the day two customers withdraw the same amount in the same hour, and it degrades from there. So every entry that touches an external rail stores that rail’s own identifier — the payment reference, the trace number, whatever the network hands back — and matching becomes a join on that field, with amount and date left as a fallback for lines that arrive without one. The other half is what you do with a break. Most breaks are timing: a genuinely pending transfer whose statement line lands tomorrow. If the process can’t tell those apart from a real discrepancy it pages somebody every night until people stop reading it, which is worse than not running it at all. So give each pending entry an expected settlement window, age the breaks against it, and escalate only the ones that outlive their window or exist on one side and not the other.
- pending → settledmodel in-flight money
- reconcileledger vs bank
- flagcatch drift
What the new pieces do
- Reconciliationservice
- Continuously matches the internal ledger against external bank/rail statements to catch any discrepancy.
- Bank Railsservice
- The outside world — banks, card networks, ACH. Slow, async, and where pending vs settled states come from.
Step 8 · The sharp edges
Precision, negatives & compliance
Money has its own footguns: floating-point rounding that leaks fractions of a cent, balances that must never go negative, multi-currency, and a mountain of audit/compliance requirements.
Store money as integer minor units (cents) or fixed-point decimals — never floats — so no cent is ever rounded away. Enforce a hard non-negative invariant (or explicit, controlled overdraft) at the check-and-debit. Handle multi-currency as separate ledgers with explicit FX conversion entries (never mix currencies in one balance). And lean on the append-only ledger for audit/compliance: it's immutable, complete, and reconstructable — exactly what auditors, regulators, and disputes require. Reversals are new compensating entries, never edits.
Why this piece earns its place
Integers stop cents leaking; they don’t stop arithmetic from producing a remainder. The moment you split a payment — a fee taken as a percentage, a payout divided between three sellers — the exact share isn’t a whole number of minor units, and if each leg is rounded on its own the legs no longer sum to the original amount. Now the entries don’t sum to zero and the invariant the whole design rests on is broken by a rounding rule, silently, on ordinary traffic. So compute splits as integers by construction: take the shares by integer division, assign the remainder deterministically to a named party, and assert that the parts add up before the transaction commits. Multi-currency has the same shape. A leg in one currency and a leg in another can’t balance against each other, so the conversion is booked through an FX account with a balanced pair on each side, and the rate used is stored on the entries themselves — a rate looked up again later is a different rate, and the books will not reproduce.
You did it
You just designed a digital wallet.
Everything you assembled, in order
- A mutable balance column is unsafe: it races, half-completes on crash, and has no audit trail.
- Store an append-only log of movements; balance is derived, never overwritten.
- Every transfer is a double-entry debit+credit in one atomic transaction — money is conserved.
- Idempotency keys make transfers exactly-once, so retries never double-charge.
- Materialized balances give O(1) reads; per-account locking prevents double-spend.
- Shard by account; cross-shard transfers use sagas/holds (reserve → confirm → compensate).
- Model pending vs settled for slow external rails; reconcile the ledger against bank statements.