Design Real-time Fraud Detection

Picture a payments company that processes card payments for a few hundred thousand online merchants. Before each payment goes to the card network, the payments service calls our risk engine and waits for an answer. The same engine also scores logins, payout changes and new card enrolments, but card payments set the hardest budget, so design for them first.

Assume 520 million payments a day, a peak of roughly four times the average during holiday evenings, and about 350 model features per decision. Fraud is rare: say 0.15% of payments are later confirmed fraudulent.

The engine has a fast loop and a slow loop. The fast loop answers one payment in tens of milliseconds: the orchestrator fans out reads to the feature store and the entity graph, runs pre-model rules, asks the model server for a score, runs post-model policy, and replies. The slow loop learns: every decision is logged together with the exact features used, outcomes are joined on later as they trickle in, and a training job produces the next model, which is backtested and then rolled out in shadow before it is trusted.

A third, middle-speed loop keeps features fresh. Every payment, login and authorisation result is published to Kafka, and a stream processor such as Flink updates rolling counters per card, account, device, IP and merchant within a second or two. The orchestrator also applies the current payment to those counters in memory before scoring, so the burst of attempts happening right now is never invisible.

Write the budget down before anything else. Every stage gets an allowance, and the orchestrator cuts a stage that overruns rather than letting the whole decision miss.

The strongest fraud signals are counts over short windows: how many cards this device has used in ten minutes, how many distinct merchants a card has tried in an hour, how much this account has spent today against its 90-day median. Card testing, where a fraudster checks hundreds of stolen numbers with tiny payments, shows up almost entirely in these counts.

The stream processor keeps each window as a ring of small time buckets, for example sixty one-minute buckets for the hour window, so sliding the window is a cheap shift instead of a recount. But a stream has lag, typically a second or two, and fraud bursts are faster than that. So the orchestrator does one more thing: after reading the stored counters, it adds the current event to them in memory before computing features. The tenth attempt in a burst therefore sees nine prior attempts from the store (or most of them) plus itself, never zero.

One stolen card is a transaction problem. Two hundred stolen cards cycled through twelve devices and three shipping addresses is a ring, and no single payment looks unusual. A batch job builds a graph where cards, accounts, devices, IPs, emails and addresses are nodes and every co-occurrence is an edge, then computes connected clusters and their history: size, age, fraud rate of members. The online path only does a single lookup: which cluster is this device in, and how bad is that cluster. Recomputing clusters every few hours is enough for most rings, and the streaming counters cover the fast attacks in between.

The classic quiet bug in fraud systems is training-serving skew. Data scientists rebuild features from the warehouse for training, but the warehouse computes "attempts in the last 10 minutes" slightly differently from the stream, or it accidentally includes events that happened after the payment. The model learns from data it will never see in production and performs worse live than it did in the notebook.

The cleanest fix is to log the exact feature vector at scoring time, alongside the decision and versions, and train on those logs. Point-in-time correctness comes free, because the vector was frozen at the moment of the decision. New features need a backfill, which the offline pipeline computes with careful as-of joins, and they enter the log as soon as they ship.

Chargebacks are the most trustworthy label and the slowest. If you train only on payments old enough to be fully labelled, the model is always three months behind the fraudsters. Mix in faster signals with care: a cardholder reporting fraud to the issuer within days, an analyst verdict, a failed challenge, an account closed for abuse. Weight these by how often they turn out right. Also remember that payments you declined never get a label at all, because they never happened. Hold back a small random slice of risky traffic, for example 0.5% of would-be declines that are sent to challenge instead, so the model keeps learning what it would have missed.

The model outputs a probability; the business needs an action. Set thresholds by cost, not by gut feel. For each payment, the expected loss of allowing it is roughly the fraud probability times the amount plus fees. The cost of declining a good payment is the lost margin and a chance of losing the customer. A challenge costs a little friction and some abandonment. The policy picks the cheapest action per band, and merchants with thin margins or high-value goods get their own cut-offs. Reason codes come from per-feature contributions computed from the trees, translated into plain phrases a reviewer can act on.