The whole design, in writing
Learn system design by building a real-time fraud detection system like Stripe Radar step by step. An interactive guide covering why static rules fail, real-time feature computation (velocity, device, geo) from a feature store, ML risk scoring blended with rules, streaming aggregations under a tight latency budget, the label feedback loop that retrains models, allow/deny/step-up decisioning with graph features, and the unhappy paths (concept drift, false positives, class imbalance, explainability).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is fraud detection?
A stolen card is being tested on your checkout right now. You have maybe 80 milliseconds to decide: let it through and eat a chargeback, or block it — and if you block a legitimate customer, you've lost a sale and annoyed a real person. Do this millions of times an hour, against attackers who adapt the moment you catch them.
A fraud detection system scores every transaction's risk in real time and decides allow / deny / challenge within a tight latency budget. The design blends fast rules with an adaptive ML model, fed by real-time features (velocity, device, geo), and closes a feedback loop so it keeps learning as fraud evolves.
What the new pieces do
- Transactionclient
- A payment or signup arriving for a real-time risk decision — allow, block, or challenge — in under ~100ms.
Step 1 · Why not just rules?
Static rules break
The first instinct: hardcode rules. "Block if amount > $2000 AND country ≠ home." It catches something on day one. Why does a rules-only system fail over time?
A hand-written rule set for fraud. Why does it fail over time?
Rules are actually fast and explainable — that's their strength. Their weakness is that they're static and gameable: fraudsters learn the thresholds and slip under them, while legitimate edge cases trip false positives.
Humans write rules easily; the problem is they can't enumerate every evolving fraud pattern, and fixed thresholds are both easy to evade and blunt (blocking real customers). You need something adaptive that learns subtle patterns.
Fraudsters probe until they find the threshold and stay just under it; meanwhile rigid limits block legitimate outliers (a real customer's big foreign purchase). Rules can't capture subtle, shifting, multi-signal patterns — that needs learning from data.
Rules alone are static and gameable: attackers probe until they find a threshold and stay just under it, while blunt limits false-positive on legitimate outliers. Fraud is a moving, subtle, multi-signal target — you can't enumerate it by hand. You need a system that learns patterns from data and adapts, while keeping rules for the clear-cut and compliance cases.
Why this piece earns its place
Keeping rules for the clear-cut cases is right, but notice what it commits you to operationally: a rule set only ever grows. Every incident review ends with a new rule, and none of them is ever deleted, because nobody can prove which ones are still earning their place. Far enough in, you have hundreds of overlapping conditions, an unknown share of your declines come from a rule written for an attack that ended long ago, and any measurement of the model is confounded by whatever the rules already removed from its traffic. So treat the rule set as code with an expiry. Attribute every hard block to the specific rule that fired and log it with the decision. Run each rule in shadow alongside its live form, so you can read what it would block if it were switched off and how much of that the model would have caught anyway. Then retire on a schedule: a rule that hasn’t fired in a long time, or whose blocks are now fully covered by the score, comes out. Otherwise the engine you kept for the obvious cases quietly becomes the thing nobody dares change.
Step 2 · The decision path
Score, then decide, fast
Put a decision service in the payment path. It has a hard constraint most systems don't: it must return a verdict in tens of milliseconds, synchronously, or it delays every checkout. What must it produce?
The decision service takes a transaction and returns a verdict — allow, deny, or step-up (challenge with 2FA/review) — within a strict latency budget. First pass: run fast rules (hard blocks, allowlists, compliance). Then compute a risk score and threshold it into a decision. It sits inline with the payment, so everything it does must be fast and reliable.
Why this piece earns its place
Putting this service inline makes its availability the payment path’s availability, so the question this step creates is what a timeout returns. There has to be a default verdict, and it has to be allow. Denying on error turns a slow dependency into an incident that looks, to every customer at once, like their card being refused — and the fraud that slips through that window costs far less than the checkout you just broke. Two things make that safe enough to say out loud. The timeout is per call, not per request: features, rules and the scorer each get their own slice of the budget, so a slow model returns a rules-only verdict instead of eating the whole window and returning nothing. And the cheap deterministic checks — blocklists, sanctions, hard compliance blocks — stay on the fail-open path, because they cost almost nothing and they are the ones you can’t afford to skip. Then alarm on the fallback rate itself. An attacker who can make you slow can make you blind, so a rising rate of timed-out decisions is an attack signal, not an infrastructure footnote.
- ~80msdecision budget
- allow/deny/step-up3 outcomes
- inlinesynchronous in checkout
What the new pieces do
- Decision Servicegateway
- Orchestrates the decision: gather features, run rules + model, and return allow / deny / step-up within the latency budget.
- Rules Engineservice
- Deterministic rules for the obvious cases (blocklists, impossible velocity) and compliance — fast, explainable, editable.
Step 3 · What the model sees
Real-time features
A single transaction — "$40, card X, IP Y" — barely tells you anything. The signal is in context: how this compares to card X's history and behavior right now. How do you get rich features within the latency budget?
A lone transaction is weak signal. Where does the fraud signal actually come from?
The signal is in context: how many transactions this card made in the last minute/hour (velocity), is this a new device, how far from the last location, account age, prior chargebacks. These are precomputed and streamed into a low-latency feature store so scoring reads them in milliseconds.
Amount and country alone are the weak, gameable signals rules already use. Real detection needs behavioral context — velocity and history — which a lone request doesn't contain.
Scanning a card's full history per request blows the latency budget. Features must be precomputed and maintained incrementally in a feature store, ready to read in milliseconds.
Compute contextual features and serve them from a low-latency feature store. The powerful signals are behavioral: velocity (transactions per card/device/IP in the last minute/hour/day), device fingerprint and whether it's new, geo distance from the last transaction, account age, and prior chargebacks. These are precomputed and updated in near-real-time, so the scorer reads them in milliseconds instead of recomputing history per request.
Why this piece earns its place
The trap here is that one feature now has two implementations — the pipeline that writes it for scoring, and whatever the training job computes when it rebuilds history from the warehouse. The day those disagree, nothing fails. Offline evaluation looks excellent, because the training job quietly held information the serving path never has: a window that includes the transaction being scored, a chargeback flag that was only set weeks afterwards, a geo distance measured against a location not yet recorded at decision time. The model learns to lean on signal that doesn’t exist in production, and its live precision comes in worse than every test promised. The fix is to stop reconstructing training data at all. Log the exact feature vector the scorer read, with its timestamp, the model version and the verdict it produced, and train on those logged vectors. It’s strictly less convenient — you can only train on features you were already computing, so a new feature costs a waiting period before it has any history — and it’s the only form of point-in-time correctness that can’t drift, because there is no second implementation left to disagree with.
- velocity#txns / window
- device + geonew? far?
- feature storems-latency reads
What the new pieces do
- Feature Computeservice
- Assembles the signals a decision needs: transaction velocity, device fingerprint, geo distance, account age, history.
- Feature Storecache
- Low-latency store of precomputed + streaming features, so scoring reads them in milliseconds instead of recomputing.
Step 4 · Score the subtle stuff
The ML model
Rules caught the obvious; features are ready. Now you need to turn dozens of signals into a single risk number that captures patterns no human wrote down — combinations like "new device + unusual amount + high velocity + far geo."
Run an ML model (often gradient-boosted trees for tabular fraud — fast and strong) that takes the features and outputs a fraud probability. Combine it with rules: a hard rule can veto, the model score handles the nuanced middle, and a threshold (tuned to your fraud-vs-friction tradeoff) maps the score to allow / step-up / deny. Keep the model fast (the latency budget) and monitored (fraud drifts, so a stale model quietly rots).
Why this piece earns its place
A scorer with dozens of features has dozens of ways to be handed nothing: a store read that spends its slice of the budget and returns late, a brand-new card with no history to aggregate, a device fingerprint that fails to resolve. What you do with the gap is a modelling decision, not plumbing. The tempting fix is to fill in a mean or a zero at serving time, and it’s the wrong one, because a mean says average and average is a confident statement. A first-ever transaction on a fresh account then scores like a typical customer — precisely the case you most wanted to look unusual. Instead, train missingness in as its own value: gradient-boosted trees handle an absent feature natively, sending it down its own branch, and you generate that absence during training at something like the rate production produces it so the model has genuinely seen it. This matters most in bulk. When the feature store degrades, nulls don’t arrive one at a time — every request loses the same signals at once, and a model that reads absence as average keeps returning comfortable scores through an entire incident.
- P(fraud)model output
- thresholdtuned tradeoff
- rules veto+ model middle
What the new pieces do
- ML Scorerdata
- A trained model that scores subtle fraud patterns rules can't express, returning a probability of fraud.
Step 5 · Keep features fresh
Streaming aggregations
Velocity features — "10 transactions on this card in the last 60 seconds" — are only useful if they reflect what happened seconds ago, including the transaction being scored. A nightly batch job is far too slow to catch a burst attack in progress. How do you keep them current?
Stream every event through a real-time processing layer (Kafka + Flink/streaming jobs) that maintains windowed aggregations — counts and sums per card/device/IP over sliding time windows — and writes them straight into the online feature store. So the moment a fraudster fires 50 rapid transactions, the velocity feature spikes and the very next decision sees it. Fresh features are what catch attacks while they happen.
Why this piece earns its place
Counting gets harder once the counter is a decision input. The stream delivers at least once, so a retried authorization, a consumer rebalance or a replay after an outage all recount the same event and the velocity feature climbs with nothing having happened. That isn’t a dented metric; it’s a customer whose own retry pushed them past a limit and got them declined. So the aggregation deduplicates on the event’s own id rather than counting arrivals, and windows are keyed by event time with a bounded allowance for late arrivals — undercount and you miss a burst, overcount and you decline someone real. The second thing worth saying out loud is that this pipeline can never see the transaction it is about to score: the event is published, travels through the aggregation, and lands in the store long after the verdict was due. So the decision service folds the in-flight transaction into the count locally before it compares against anything, and the stream supplies everything up to it. Otherwise the first transaction of a burst is always judged against a window that doesn’t contain it.
- sliding windows1m / 1h / 1d
- secondsfeature freshness
- catch in flightburst attacks spike velocity
What the new pieces do
- Event Streambus
- Every event flows here so real-time aggregations (velocity, counts per card/device) stay fresh for the next decision.
Step 6 · Learn from the outcome
The label feedback loop
The model was trained on yesterday's fraud, but fraud evolves — a model left alone silently decays. To keep learning, you need ground truth: which transactions actually turned out fraudulent. But that truth arrives late (a chargeback can take weeks). How do you close the loop?
Collect labels from every source of truth — confirmed chargebacks, analyst verdicts from the review queue, customer reports — and feed them back to retrain the model regularly on fresh fraud patterns. Handle the label delay (chargebacks lag) by using faster proxy signals where possible and being deliberate about training windows. Continuously monitor live metrics (approval rate, fraud rate, score distribution) to catch drift before the slow labels confirm it.
Why this piece earns its place
There’s a bias baked into this loop that no amount of retraining repairs: you only ever learn the outcome of a transaction you allowed. Everything the system denied has no label, because nothing could happen to it — no chargeback, no complaint, no confirmation either way. So each retrain is fitted to the population the previous model approved, and the model grows steadily more confident about a region of the space it stopped sampling. A boundary that was slightly too aggressive on some legitimate segment gets encoded as fact, and the evidence that would contradict it can no longer be produced. The defense is to buy the missing labels deliberately: let a small randomized slice of transactions near the boundary through despite their score, and treat what happens to them as the unbiased sample the rest of your data isn’t. That costs real fraud losses, so you cap it by amount and keep it away from the clearly-bad tail. But it’s the only measurement that can tell you whether the boundary is still in the right place, and declining to pay for it just means never finding out.
- chargebacksground truth (weeks late)
- retrainon fresh fraud
- monitordrift before labels
What the new pieces do
- Labelsdata
- Ground truth — confirmed fraud, chargebacks, analyst verdicts — fed back to retrain the model as fraud evolves.
Step 7 · The verdict & the network
Decisioning, step-up & graphs
A binary allow/deny is crude — the borderline cases deserve a middle path — and fraud is rarely a lone transaction: it's rings of linked accounts, shared devices, and reused cards. How do you decide well and see the network?
The model scores a transaction at 0.52 fraud probability, squarely in the uncertain middle. What should the decision be?
A hard binary cutoff treats a coin-flip-confidence score with the same certainty as a 0.99 or a 0.01 — you either eat the fraud or needlessly block a real customer on a case the model itself is unsure about.
The uncertain middle is exactly where added friction pays off: a real customer clears 2FA in seconds while a fraudster often cannot, and the outcome becomes a high-quality label on a case the model was genuinely unsure about.
That converts every ambiguous legitimate transaction into a lost sale, which is far more costly than a brief step-up challenge — deny-by-default is safe for the business but hostile to real customers.
Use a three-way decision: auto-allow the clearly-good, hard-deny the clearly-bad, and step up the uncertain middle (2FA challenge or a human review queue) — turning risk into friction only where it's warranted, and generating labels. Add graph features: link accounts by shared device/card/IP/address to spot fraud rings a per-transaction view misses (one flagged account taints its cluster). Reviews and challenges feed back as high-quality labels.
Why this piece earns its place
Step-up reads like a free middle path and it isn’t — it’s a rate you are spending. Some share of challenged customers abandon instead of completing, the channel carrying the challenge has its own cost and its own delivery failures, and all of that lands on the population you were unsure about, which skews genuine. So the number to manage isn’t really the threshold, it’s the fraction of traffic you’re willing to challenge: fix that as a budget, rank by score, and let the thresholds float to fill it. A noisy day then spends the same friction on the riskiest cases instead of quietly challenging far more people. Be careful, too, about what you conclude from a pass. A completed challenge is a good signal, not a clean one — challenges get defeated, and an attacker who also controls the phone clears it — so treating every pass as confirmed-legitimate feeds the retraining loop the one label an adversary can manufacture on demand. Weight it below a chargeback, and watch for accounts that clear challenges suspiciously often.
- allow / step-up / deny3-way
- graph featurescatch rings
- reviews→ labels
What the new pieces do
- Review Queuedata
- Borderline cases routed to step-up auth (2FA) or human analysts, whose verdicts become high-quality labels.
Step 8 · The sharp edges
Drift, false positives & imbalance
Fraud detection has vicious edges: an adversary actively adapts (concept drift), false positives cost real revenue and trust, fraud is a tiny fraction of traffic (class imbalance), and you may have to explain a decline to a regulator.
Fight drift with frequent retraining, drift monitoring, and champion/challenger model tests. Take false positives seriously — measure the cost of blocking good customers, and prefer step-up over hard deny near the boundary. Handle class imbalance (fraud is <1%) with proper metrics (precision/recall, PR-AUC — never raw accuracy), resampling, and cost-sensitive training. Ensure explainability (reason codes per decision) for disputes and compliance, and defend the pipeline from adversarial probing and gaming.
Why this piece earns its place
Explainability is the requirement that quietly arms the adversary. A reason code exists so a customer can be told why and a regulator can be answered — and it is also a free read of your decision boundary. Tell someone their decline was driven by a new device and high velocity and you have named the two things to change before the next attempt. Attempts are not independent events; a run of declines from the same actor is someone reading your responses one at a time. So split the artifact in two. Externally, a small fixed vocabulary of coarse, stable categories, chosen so each code covers a wide range of underlying causes and so the wording doesn’t shift every time the model is retrained. Internally, the full decision record — model version, the feature vector that was read, which rules fired, the score and the threshold in force — kept for disputes, audits and the next retrain. And treat the probing itself as signal rather than noise: the sequence of attempts on a card or device belongs in the features, so a patient attacker walking the boundary looks less safe with each try, not more.
You did it
You just designed a fraud detection system.
Everything you assembled, in order
- Static rules are gameable and blunt — fraud is adaptive and subtle, so it needs learning.
- A decision service returns allow/deny/step-up inline, under a tight (~80ms) latency budget.
- Real-time contextual features (velocity, device, geo, history) from a feature store carry the signal.
- An ML model scores the subtle patterns; rules veto the obvious; a tuned threshold decides.
- Streaming aggregations keep velocity features fresh enough to catch attacks in flight.
- A label feedback loop (chargebacks, reviews) retrains the model as fraud evolves.
- Three-way decisioning + graph features + reason codes handle nuance, rings, and compliance.