Design Crypto Exchange

Assume 15M verified users, 1.5M of them active on a typical day, 400 markets, and 40 blockchains. Trading is very uneven: the top ten markets carry most of the volume, and market makers send far more orders than humans.

Two halves meet at the ledger. On the trading side, the order service authenticates, validates the order and places a hold on the funds it could consume. Only then does the order go to the sequencer for its market, which stamps it with a sequence number and appends it to a replicated journal. The matching engine consumes the journal in order, updates its in-memory book and emits fills, which go back into the journal as events.

Downstream consumers read the journal and never talk to the engine directly. The ledger turns fills into double-entry postings and releases or converts holds. The market data publisher turns book changes into depth snapshots, trade ticks and candles. Because everything downstream is driven by the same ordered journal, any consumer can be rebuilt by replaying it.

On the custody side, a chain watcher per network runs its own nodes, follows new blocks, matches outputs to customer deposit addresses and waits for the network's confirmation rule before crediting. Withdrawals go the other way: screening, a policy check, a signature from keys held in hardware or split across parties, then broadcast and tracking until confirmed. Hot wallets hold only a small float; warm and cold wallets refill them through a slower, human-approved process.

Each market gets one engine thread with the full order book in memory. Bids and asks are sorted maps keyed by price, each price level holds a FIFO queue of orders, and a hash map from order id to its queue node makes cancels constant time. A thread per market means no locks inside the engine, and since one market's orders never interact with another's, markets scale out by placing engines on different cores and hosts.

Determinism is the property that makes it safe. The engine reads inputs only from its journal, never consults the clock or random numbers, and never does network or disk I/O while matching. Given the same journal, it always produces the same fills. That gives you crash recovery (reload the last snapshot, replay the journal tail) and hot standby (a second engine on another host consumes the same journal and stays a few microseconds behind, ready to take over).

Prices and sizes are fixed-point integers internally, scaled by the market's tick size and lot size. Floating point is never acceptable for money: 0.1 plus 0.2 does not equal 0.3 in binary floating point, and an exchange cannot round a customer's balance by accident.

The sequencer for a market is the single writer of its journal. It assigns a monotonically increasing sequence number to every input (new order, cancel, admin halt) and replicates it to at least two other machines in different zones before the engine processes it. A Raft group per market or a primary with synchronous backups both work; what matters is that an input is durable on a quorum before the engine acts on it.

Outputs follow the same rule. Fills and book changes are written to the output journal, and only events in the journal are visible to the ledger and to traders. If the engine crashes after matching but before writing, the standby replays the same input and produces the same fill, so nothing is lost and nothing is doubled. Snapshots of the book every few minutes keep replay times short.

A user with 10,000 USDT can send two buy orders for 8,000 USDT each in the same millisecond, to two different markets. If each engine checked the balance itself, both could pass. The fix is to move the balance check before the engine. The order service atomically moves the order's maximum cost from available to on_hold in the user's balance row, conditional on available being high enough. The second order fails that condition and is rejected before it reaches any book.

Fills then convert holds into transfers: the buyer's held USDT goes to the seller, the seller's held ETH goes to the buyer, minus fees, all as one ledger transaction. Cancels and expired orders release what is left of the hold. For market orders with no price limit, hold the full quote amount the user specifies or a size-based cap with a protection price. Balance rows are sharded by user, so holds for different users never contend.

Each network has a watcher service connected to the exchange's own full nodes, never only to third-party APIs. It follows the chain tip, decodes each block, and looks for outputs paying any of millions of deposit addresses, using a Bloom filter in front of the address table to keep the per-block check cheap. A match creates a pending deposit with zero confirmations.

Crediting waits for the network's rule. Proof-of-work chains need several blocks on top because a reorganisation can replace recent blocks and erase a transaction; chains with explicit finality can be credited when the block is finalized. If a reorg drops a pending deposit, the watcher marks it orphaned. For large amounts on weaker chains, require extra confirmations, and let users trade a credited-but-recent deposit while still blocking its withdrawal until it is deeply buried.

Tokens need care too. A transfer event from a fake contract with the same symbol is worthless. Watchers credit only events from an allow-listed contract address per asset, and they check the actual transfer amount, not what an event claims.

A withdrawal first places a hold, then runs checks: new device or new address, an address book cooling-off period after changes, velocity limits, sanctions screening of the destination, and comparison with the user's normal behaviour. Requests that pass go to the signing service, which holds keys in HSMs or splits them with multi-party computation so no single machine ever has the full key. Policy limits live inside the signer itself, so even a compromised application server cannot sign a transfer above the hourly cap.

Hot wallets carry only enough to cover a few hours of normal withdrawals. A refill job tops them up from warm wallets with multi-person approval, and warm wallets are refilled from cold storage on a slower schedule. If the hot wallet is drained, the loss is bounded to that float. Exchanges also publish proof of reserves: a Merkle tree of customer balances, where each user can verify their own leaf, plus signed on-chain evidence that the exchange controls enough assets to cover the total.