Design a Fraud Detection System: Real-Time ML System Design

The question: design a system that decides, in real time, whether a card payment is fraudulent before it's authorised. Missing fraud costs money and trust; blocking good customers costs revenue and goodwill. Labels arrive weeks late, fraudsters adapt, and positives are rare. It's a favourite ML design question at payments companies, banks and marketplaces because it tests all of that at once.

1. Clarify requirements

2. Frame it as an ML problem

Binary classification: p(fraud | transaction, history). The decision is a cost-based threshold, not 0.5: decline when the expected loss of approving exceeds the expected loss of declining. Different thresholds for amount bands and merchant types are common.

expected loss if approved = p(fraud) × amount
expected loss if declined = (1 − p(fraud)) × margin_lost × churn_factor
decline when p × amount > (1 − p) × margin × churn
→ a $2,000 laptop needs a much lower p to decline than a $5 coffee

3. Data and labels

• Rows: every scored transaction with the features used at decision time (logged, to avoid skew).
• Label delay: chargebacks take weeks, so training data is 2-3 months old, and recent labels are incomplete. Train on matured windows; use review outcomes and confirmed reports as faster partial labels.
• Selection bias: declined transactions never get a chargeback, so you can't tell if they were fraud. Let a tiny random sample of would-be declines through (or step them up) to learn; see feedback loops.
• Imbalance: 0.1% positives; weight by amount and use PR-AUC. See training data.

4. Architecture

5. Features

Velocity features must be fresh to the second: a fraudster testing a stolen card fires many small transactions quickly. They come from a stream processor writing to a low-latency store. See feature engineering and feature stores.

6. Models

• Rules first: sanctions lists, known-bad devices and hard limits. Fast to change when an attack starts; analysts own them.
• Gradient-boosted trees on tabular features: accurate, fast on CPU, and they give feature contributions for explanations to reviewers.
• Optional deep or graph models for sequences of transactions and fraud rings, often as features into the tree model.
• Anomaly detection (unsupervised) to flag new patterns nobody has labelled yet.

7. Latency and reliability

Budget (100 ms total, inside authorisation):
  feature fetch (parallel)  15 ms   online store p99
  rules                      5 ms
  model                     15 ms   trees on CPU
  decision + logging         5 ms   logging async
  network + headroom        60 ms
Fallback: if the model or features time out → rules-only decision with a
conservative threshold for high amounts; never block all traffic on an outage

The fraud service is in the payment path, so it gets the same treatment as any critical dependency: timeouts, a circuit breaker and a fallback.

8. Human review loop

• Mid-score transactions go to analysts (or are held briefly); queue size is set by reviewer capacity.
• The queue is ordered by expected loss (p × amount), not by arrival time.
• Reviewer decisions are fast labels and feed retraining; their disagreement rates are tracked.
• Customers' disputes and "this was me" confirmations are labels too.

9. Evaluation

Online A/B tests are delicate: a control arm with a weaker model loses real money. Use small allocations, shadow scoring of the challenger, and backtests on matured data. See ML evaluation.

10. Adapting to adversaries

• Monitor score distributions and decline rates per merchant and BIN range hourly; a sudden shift often signals a new attack.
• Retrain regularly on matured labels, and let analysts push rules within minutes in between; see continual learning.
• Keep some features secret and avoid exposing exact decline reasons to the requester.