Financial & Trading
Design Payment Gateway
01
The problem
The problem
Build the service that merchants integrate to accept card payments online. A merchant's server creates a payment, the customer enters card details in a hosted field, and the gateway asks a card processor or acquiring bank to authorise the amount. Later the merchant captures the funds, perhaps issues a refund, and the gateway settles money into the merchant's bank account, minus fees.
Most system design problems tolerate a little staleness or an occasional duplicate. This one does not. Correctness beats latency, and money must balance. A duplicate charge is a support ticket and a chargeback; a lost charge is revenue the merchant never sees; and a ledger that does not match the bank is an audit finding.
A request to the processor can time out with no reply. The charge may or may not have happened, and you must find out before retrying.
The bank's settlement file, not your database, is the final word on what moved. You reconcile against it every day.
Raw card numbers bring heavy compliance obligations. The less of your system that ever sees one, the better.
A payment lives for months: authorised, captured, partially refunded, disputed. Every step must be recorded and auditable.
02
What the gateway must do
What the gateway must do
- Merchants create a payment for an amount and currency and confirm it with a tokenised card.
- Support authorise-then-capture (hotels, marketplaces) and immediate capture (digital goods).
- Full and partial refunds, voids of uncaptured authorisations, and dispute handling.
- Signed webhooks to the merchant for every state change.
- Daily payouts to merchants net of fees, with reports that tie every payout to its payments.
- Route to more than one processor and fail over between them.
- No double charges, ever, even when merchants retry, workers crash or processors time out.
- Every cent is accounted for in a double-entry ledger that balances at all times.
- Payment API availability of 99.99%; p99 for confirmation under 2 seconds, most of it the bank's time.
- Card data handled within a minimal PCI DSS scope.
- Full audit history: nothing is updated in place without a record of what it was before.
Clarify whether the gateway is the merchant of record holding funds before payout, or only passes requests to a processor that settles directly with merchants. Holding funds means you also need a ledger, payout scheduling and reserves. Most interviewers want that version, so confirm it before you draw.
03
Sizing the load
Sizing the load
Assume 25 million payments a day across 400,000 merchants, with checkout traffic peaking at about eight times average during a large sale event. Each payment produces several ledger entries and events over its lifetime.
| Quantity | Working | Result |
|---|---|---|
| Average payment rate | 25M / 86,400 s | about 290 / s |
| Peak payment rate | 8x during a sale | about 2,300 / s |
| API calls per payment | create, confirm, capture, retrieve, plus retries | about 6 |
| Peak API load | 2,300 x 6 | about 14K requests / s |
| Ledger entries | 25M x about 6 lines (auth, capture, fee, payout) | 150M rows a day |
| Payment rows | 25M x 365 x 2 KB with history | about 18 TB a year |
| Webhooks | 25M x 3 events | 75M a day, about 900 / s |
| Processor latency | network authorisation round trip | 300 ms to 1.5 s |
| In-flight authorisations | 2,300 / s x 1.5 s worst case | about 3,500 concurrent calls |
| Daily volume | 25M x $42 average | about $1B a day |
A few thousand writes a second fits a well-sharded relational database comfortably. The design effort goes into correctness under failure, not raw scale. With a billion dollars a day flowing, a 0.01 percent duplicate rate is a hundred thousand dollars of wrong charges daily.
04
API design
API design
Always store and transmit money in the currency's smallest unit as an integer. Floating point cannot represent 0.10 exactly, and rounding errors in a ledger are the kind of bug you find in an audit, months later.
05
Architecture overview
Architecture overview
flowchart LR
B["Customer browser"] --> HF["Hosted card field"]
HF --> V[("Card vault, PCI zone")]
M["Merchant server"] --> API["Payments API"]
API --> IK[("Idempotency records")]
API --> PS["Payment service: state machine"]
PS --> PDB[("Payments DB with outbox")]
PS --> RT["Processor router"]
RT --> V
RT --> P1["Processor A"]
RT --> P2["Processor B"]
PDB -. "outbox relay" .-> EV[["Event log"]]
EV --> LG[("Double-entry ledger")]
EV --> WH["Webhook sender"]
REC["Reconciliation jobs"] --> LGCard details never touch the merchant's servers or most of ours. The customer types them into a hosted field served from our domain, which posts directly to the vault, a small, separately audited service that encrypts the number and returns a token. Everything else in the system works with tokens. Only the processor router, inside the same compliance zone, asks the vault to detokenise when it builds a request to a processor.
The payment service owns a state machine per payment. Every transition, such as authorizing to authorized, is a database write that checks the current state, so two concurrent requests cannot both move the same payment. In the same transaction, the service writes an event row to an outbox table. A relay reads the outbox and publishes events to a log, which the ledger, webhooks and analytics consume. This avoids the classic bug where the database commits but the event publish fails, or the reverse.
Reconciliation jobs compare three views of each day: our payment records, our ledger, and the processors' settlement files. Any mismatch becomes a case for an operator.
| Component | Owns | Store |
|---|---|---|
| Card vault | encrypted card numbers, tokens | HSM-backed keys, isolated DB |
| Payment service | payment state, attempts, outbox | Postgres sharded by merchant |
| Idempotency records | key to stored response, 24 h | same DB as payments |
| Ledger | immutable debit and credit lines | append-only relational tables |
| Processor router | routing rules, health, retries | stateless, config in DB |
| Webhook sender | delivery attempts per endpoint | queue plus attempts table |
06
Deep dive: charging exactly once
Deep dive: charging exactly once
A customer is charged exactly once for each payment the merchant intended. Networks cannot give you exactly-once delivery, so you build it from at-least-once retries plus idempotency at every boundary, and a way to resolve outcomes you did not see.
Idempotency at the API
Merchants retry. Their HTTP client times out at 10 seconds while our processor call takes 11, or their load balancer replays a request. Every mutating call therefore carries an idempotency key chosen by the merchant. On first sight of a key, the API inserts a record with status started and a hash of the request body, inside the same database as the payment. When the work finishes, it stores the response on that record.
A second request with the same key gets one of three answers. If the record is finished, return the stored response byte for byte. If it is still in progress, return 409 and tell the client to retry shortly. If the body hash differs, return an error, because the merchant is reusing a key for a different payment, which is a bug on their side worth surfacing loudly.
flowchart TD
R["Confirm request with idempotency key"] --> K{"Key seen before?"}
K -->|finished| SAME["Return stored response"]
K -->|new| ST["Move payment to authorizing, write attempt id"]
ST --> CALL["Call processor with attempt id as its reference"]
CALL --> RES{"Reply?"}
RES -->|approved| AU["Move to authorized, outbox event"]
RES -->|declined| DE["Move to declined, outbox event"]
RES -->|timeout| UNK["Stay in authorizing, mark unknown"]
UNK --> LOOK["Resolver queries processor by attempt id"]
LOOK --> AU
LOOK --> DE
AU --> SAVE["Store response on idempotency record"]
DE --> SAVEThe timeout problem
The dangerous moment is when we send an authorisation and hear nothing back. The bank may have approved it, declined it or never received it. Retrying blindly risks a second hold on the customer's card; giving up risks failing a payment that actually succeeded.
The answer is to give every attempt its own ID, written to our database before the processor call, and pass it to the processor as the merchant reference. If the call times out, the payment stays in authorizing with an unknown outcome. The API tells the merchant the payment is processing. A resolver job then asks the processor for the status of that reference, with backoff, until it gets a definite answer, and completes the transition. Most processors also accept the reference as their own idempotency key, so a retry with the same attempt ID cannot create a second authorisation.
If the processor still cannot say after a long window, the system voids the reference to release any hold and fails the payment. Better to ask the customer to pay again than to risk charging twice.
A state machine with guarded transitions
Each payment moves through a small set of states: requires_payment_method, authorizing, authorized, captured, partially_refunded, refunded, declined, voided. Every update is a conditional write such as "set status to captured where id is X and status is authorized and version is 7". If two workers race, one of them updates zero rows and backs off. There are no in-place edits without a recorded transition, which makes the history auditable by construction.
Double-entry ledger
The payments table tells you the state of each payment. The ledger tells you where the money is. Every movement is recorded as at least two lines that sum to zero: a capture of 2,499.00 debits "card network receivable" and credits "merchant balance", while the fee debits the merchant and credits "gateway revenue". Lines are append only; a refund is new lines in the opposite direction, never an edit. Because every entry balances, the sum of all accounts is always zero, and any non-zero total means a bug you can detect immediately.
The ledger is written from the event log, keyed by event ID with a unique constraint, so replaying events never double-books.
Reconciliation closes the loop
Each morning, processors deliver settlement files listing what they actually moved. A job matches each line to our records by reference and amount. Three kinds of mismatch appear: lines the processor has but we do not (often the result of a timeout we never resolved), lines we have but they do not, and amount differences from currency conversion or fees. Each one becomes a case with an owner. Reconciliation is how you find the bugs that idempotency missed.
"Merchants send an idempotency key that maps to a stored response. Each processor attempt gets an ID that is saved before the call and sent as the processor reference, so timeouts are resolved by asking, not by retrying blind. Payments move through guarded state transitions with an outbox in the same transaction. A double-entry, append-only ledger tracks money, and daily reconciliation against settlement files catches whatever slipped through."
07
Decisions and what they cost
Decisions and what they cost
Transactions, unique constraints and conditional updates are exactly the tools a state machine and an idempotency table need. Shard by merchant to scale writes.
Correctness firstScales effortlessly, but two replicas accepting different transitions for the same payment is the precise failure you cannot allow. Fine for analytics copies, not for the record.
Wrong guaranteesThe event row commits with the state change or not at all. A relay publishes it later, so downstream consumers see every transition exactly as the database does, at the cost of a little extra delay.
No lost eventsIf the process dies between the two, either the ledger misses a capture or it books one that was rolled back. Both show up only at reconciliation.
Dual write gapFail over when one processor degrades, route by card type or region for better approval rates, and negotiate fees. Each processor needs its own integration and reconciliation.
Resilience and costMuch simpler to build and reconcile. Its outage is your outage, and you have no leverage on pricing.
Fine early onA model scores each payment in a few tens of milliseconds and blocks or challenges the risky ones with 3-D Secure. Slower checks run after authorisation and can void before capture.
BalancedCatches more before the bank call, but adds hundreds of milliseconds to every checkout and turns any fraud service outage into a payment outage.
Too slow, too coupled08
Failure modes
Failure modes
The bank approved the authorisation but the reply was lost. A naive retry places a second hold on the customer's card.
Persist the attempt ID before calling, send it as the processor reference, keep the payment in a pending state on timeout, and resolve it by querying the processor before any retry.The merchant's code generates a fresh idempotency key on every retry, so our dedup never fires and the customer is charged twice.
Document that keys must be stable per intended payment, and add a soft guard: flag or block a second payment with the same merchant reference, card and amount within a short window.The morning file shows 14 captures we have no record of, all from one hour when a database failover happened.
Reconciliation raises cases automatically; operators match each line to an unresolved attempt and the resolver completes the transition, with ledger entries written from the resulting events.Approval rates on one processor drop from 92 percent to 60 percent and latency triples, while traffic is at its peak.
Track approval rate and latency per processor in real time and shift new payments to the healthy one. Never re-send an already attempted payment to a second processor without first voiding the first attempt.Events fail for a day. When the endpoint returns, it receives thousands at once, some out of order.
Retry with exponential backoff for several days, include event IDs and timestamps so the merchant can deduplicate and order them, and offer an events API to fetch anything missed.A debug log line prints a full request body containing card data, which drags the whole logging stack into compliance scope.
Keep raw card data inside the vault only, tokenise at the edge, and scrub logs with pattern detectors that redact anything shaped like a card number.09
Mistakes to avoid
Mistakes to avoid
The first attempt may have succeeded. A retry creates a second hold or a second charge on the same card.
Binary floats cannot represent most decimal fractions, so sums drift by fractions of a cent and ledgers stop balancing.
There is no history, concurrent updates can lose writes, and you cannot explain to an auditor how the number got there.
Every service, log and backup that sees a card number falls into compliance scope and becomes a breach risk.
10
How to handle the interview
How to handle the interview
State the invariant first. "No customer is charged twice, and every cent balances." Then tie each component back to that sentence.
Walk through the timeout case slowly. This is the question interviewers are waiting for. Attempt ID before the call, unknown state, resolve by asking.
Explain the idempotency record precisely. Where it is stored, what happens on a concurrent duplicate, and why the body hash matters.
Use a state machine with conditional writes. Name the states and show one guarded transition.
Draw the ledger as double entry. One capture example with two lines makes the concept concrete in seconds.
Mention the outbox. It shows you know the dual write problem and how to avoid it without distributed transactions.
Shrink PCI scope. Hosted fields, a vault and tokens. Interviewers notice when candidates keep card data out of their main services.
End with reconciliation. Admitting that software has bugs and showing how you would catch them daily is a mature finish.
11
Growing the design in stages
Growing the design in stages
The app calls a processor directly and stores a payment row. Card data never touches your servers because the processor's hosted page collects it.
Merchant retries cause the first double charges, so idempotency keys and guarded transitions arrive, along with signed webhooks.
Once you hold money before paying out, a double-entry ledger, payout schedules and daily matching against settlement files become mandatory.
A dedicated vault shrinks compliance scope, the outbox removes dual writes, and a router spreads traffic across processors. The single-processor dependency is gone.
Local payment methods such as UPI, bank transfers and wallets join cards, routing optimises approval rates per country, and data residency rules shape where records live.
Watch and read
Try next
Free to read · better with landed
Whiteboard this with landed
landed runs it as a live system design round on the whiteboard and grades your trade-offs.