Financial & Trading
Design Bidding Platform
01
Problem Statement
Problem Statement
Design a platform supporting millions of simultaneous timed auctions. Sellers list items with a starting price, optional reserve, and end time. Buyers place bids in real-time. The system supports proxy bidding (automatic counter-bids up to a secret maximum) and anti-sniping (time extensions when bids arrive near the deadline).
The core tension: the system must be both blindingly fast (sub-200ms bid acknowledgment) and absolutely correct (the wrong person must never win an auction). In most distributed systems you pick availability and tolerate stale reads. Here, for the bid-write path, you need linearizable consistency — and you need it at low latency.
Core question: How do you serialize millions of concurrent bids across 50M auctions — especially when a single hot auction receives 50+ bids/second in its final seconds — without sacrificing speed, correctness, or durability?
02
Requirements
Requirements
Functional Requirements
- Auction lifecycle: Sellers create listings with start price, reserve price (hidden), bid increment, and end time. States: DRAFT → SCHEDULED → ACTIVE → CLOSING → CLOSED → SETTLED.
- Bid placement: Buyers place bids that must exceed current highest + minimum increment. Response is immediate and unambiguous: "You are the highest bidder" or "You've been outbid."
- Proxy bidding: Buyer sets a secret max. System auto-bids the minimum needed. Two competing proxies resolve instantly via a mathematical Vickrey-like formula — no iterative loop.
- Anti-sniping: Configurable per-auction. Bids in the final N minutes extend the auction by N more minutes, up to a max number of extensions.
- Real-time updates: All watchers see price changes within 1-2 seconds via WebSocket push.
- Auction closure: Precise to within 1 second. In-flight bids accepted via a 2-second grace window using API Gateway timestamps.
- Settlement: Payment hold → shipment → delivery confirmation → capture → seller payout. Full dispute and chargeback handling.
- Audit trail: Immutable, append-only bid log with hash-chained event records for regulatory compliance.
Non-Functional Requirements
- Linearizable consistency on the bid-write path — no stale reads, no lost updates, no double-acceptance.
- Sub-200ms bid acknowledgment latency end-to-end.
- 99.99% availability for bid placement (~52 min downtime/year).
- Handle 100x traffic spikes during auction sniping surges.
- Auction closure within 1 second of scheduled end time.
- Zero bid loss — acknowledged bids must survive server crashes.
- Complete auditability for dispute resolution and regulatory compliance.
03
Scale Estimation
Scale Estimation
Derived from eBay-scale assumptions: 150M registered users, 50M MAU, 15M DAU. Only ~10% of DAU are active bidders.
Key Derivation
500K auctions closing/day × 8 avg bids = 4M bids/day → ~50/sec average. The critical number is the per-auction peak: a hot auction with 50 bids in 5 seconds = 10-50 bids/sec on a single Redis key. Each Lua script executes in ~0.5ms, so 50 bids/sec consumes only 25ms of Redis CPU — well within capacity. The real bottleneck is fan-out: 50 bids/sec × 50K watchers = 2.5M push messages/sec for one auction.
04
API Design
API Design
POST /v1/auctions/{auction_id}/bids
Authorization: Bearer <token>
Idempotency-Key: "bid_8f14e45f"
{ "amount": 79000, "type": "MANUAL" }
→ 200 OK
{
"bid_id": "bid_Qm3nR8xP",
"status": "ACCEPTED",
"outcome": "OUTBID", // proxy countered instantly
"your_bid": 79000,
"current_price": 79500, // proxy's counter-bid
"bid_count": 26,
"anti_snipe": { "extensions_used": 1, "exhausted": false }
}
→ 409 Conflict (bid too low — concurrency signal, not validation error)
→ 503 Retry (auction migrating or Redis failover)CONNECT wss://ws.bidplatform.com/v1/auctions/{auction_id}
← { "type": "BID_UPDATE", "current_price": 79500,
"bid_count": 26, "your_status": "OUTBID",
"time_remaining_seconds": 258, "reserve_met": false }
← { "type": "TIME_EXTENDED", "new_end_time": "...",
"extension_number": 1, "max_extensions": 3 }
← { "type": "AUCTION_CLOSED", "final_price": 95000,
"winner": "u***K", "you_won": true }Design principle: REST for actions (idempotent, structured errors), WebSocket for observations (real-time, read-only). Bids are never placed via WebSocket — no idempotency guarantees on fire-and-forget channels. The Idempotency-Key is mandatory — sniping users will retry frantically on timeouts.
05
High-Level Architecture
High-Level Architecture
Seven layers, each with a clear responsibility. The critical insight: the bid-write path is synchronous and in-memory (Redis Lua), while everything else is async via Kafka. This split — strong consistency on writes, eventual consistency on reads — is the fundamental architectural decision.
06
Deep Dive — Auction Engine Serialization
Deep Dive — Auction Engine Serialization
The Redis Lua script is the heart of the system. Every bid — manual or proxy — is processed atomically in a single Lua execution. Redis's single-threaded model guarantees linearizability by construction: no locks, no retries, no race conditions. The script validates, accepts, resolves proxies, checks anti-snipe, and returns the outcome in ~0.5ms.
Why Naive Approaches Fail
PostgreSQL Optimistic
UPDATE WHERE current_price = expected. Under contention, retry storms: bid N+1 retries after N succeeds. At 50 bids/sec, latency explodes.
SELECT FOR UPDATE
Row lock held for 5-20ms (network + fsync). 50th bid waits 500ms in queue. App crash holds lock for 30 seconds.
The Lua Script Pipeline
The script executes in 7 steps, all within a single atomic Redis call:
1. Read state — HGETALL auction state (~0.1ms). 2. Validate — auction active, not expired, not seller, not already highest, amount ≥ min. 3. Accept bid — update price, highest bidder, increment count. 4. Resolve proxies — mathematical resolution: winner pays min(loser_max + increment, winner_max). No iterative loop. 5. Anti-snipe — if within window and extensions remaining, extend end_time. 6. Write state — HMSET updated fields (~0.1ms). 7. Return — JSON result with outcome, price, flags.
Proxy Resolution — The Vickrey Formula
When two proxies compete (Alice max $500, Bob max $350), the result is computed in one step: Alice wins at min($350 + increment, $500) = $360. No looping through 35 counter-bids. Tie-breaking uses a composite score encoding max × 10¹³ + (10¹³ − timestamp), so the earlier proxy wins ties.
sequenceDiagram
participant C as Client
participant GW as API Gateway
participant BS as Bid Service
participant R as Redis (Lua)
participant K as Kafka
participant WS as WebSocket Tier
C->>GW: POST /bids (amount=$500)
GW->>GW: Auth + Rate Limit
GW->>BS: Forward + X-Received-At
BS->>BS: Idempotency check
BS->>R: EVALSHA bid_processor.lua
R->>R: Validate → Accept → Proxy resolve → Anti-snipe
R-->>BS: {ok:true, outcome:OUTBID, price:510}
BS-->>C: 200 OK (45ms total)
BS->>K: Publish BID_ACCEPTED
K->>WS: Consumer reads event
WS->>WS: Personalize per-connection
WS-->>C: Push: BID_UPDATE (80ms)
Critical Constraint: Script Must Complete in <2ms
Redis is single-threaded. While the Lua script runs, ALL other commands on that shard are queued — including bids for other auctions. At 0.5ms average with 50 ops/sec on a hot key, we consume only 25ms/sec of Redis time. The sorted set operations (proxy lookup) are O(log N) due to skip list internals. Monitoring: p99 script time > 1ms triggers an alert; > 2ms triggers investigation.
07
Key Design Decisions & Tradeoffs
Key Design Decisions & Tradeoffs
Bid Serialization
Redis Lua Script
Single-threaded atomicity. 0.3-0.8ms per bid. No locks, no retries. 20-50x faster than DB. Cost: no durability — mitigated by sync replication + Kafka + PG snapshots.
PostgreSQL Row Locks
Full ACID. 5-20ms per lock (disk fsync). At 50 bids/sec on one auction, last bid waits 500ms+. Lock held during app crash for 30s. Correct but too slow.
Auction Closure Strategy
Hard Close + Gateway Timestamp Grace
Auction stops accepting bids at end_time. Bids that entered the Gateway before end_time are accepted within a 2-second window. Clean, fair, legally defensible.
Pure Hard Close (No Grace)
Simpler. eBay's approach. But penalizes bidders with higher network latency. A bidder in Mumbai is disadvantaged vs. one near the data center.
Async Event Bus
Kafka for All Async Processing
Decoupled consumers. Event replay on failure. Ordered by auction_id partition. Adding new consumers (analytics, ML) requires zero changes to Bid Service.
Direct HTTP Service-to-Service
Lower latency (2-5ms vs 5-15ms). Simpler topology. But tight coupling: every new consumer requires Bid Service changes. No replay. No buffering during outages.
Read/Write Consistency Split
CQRS — Strong Writes, Eventual Reads
Writes serialized through Redis master. Reads from replicas, caches, Elasticsearch. 1-2 second staleness on reads. Scales reads independently to 1000x write volume.
All Reads from Authoritative Source
Perfectly consistent reads. But 50K reads/sec per popular auction would overwhelm the Redis master. Can't scale reads without scaling the serialization point.
08
What Can Go Wrong
What Can Go Wrong
Redis Master Crash During Hot Auction
All in-memory state for ~500K auctions on that shard is lost. Mitigation: Sentinel failover to sync replica in 2-5s. Sub-50ms data loss window. In-flight bids get 503, clients retry with same idempotency key. Kafka event log enables state reconstruction.
Ghost Bid — Server Crash After Lua, Before Kafka Publish
User told "bid accepted" but event never reaches PostgreSQL or notifications. Mitigation: Bid receipt written inside Lua script (same Redis transaction). Retry checks receipt before re-processing. Reconciliation job detects Redis/PG divergence every 5 minutes.
Scheduler Leader Crash at Peak Closure Time
Auctions stop closing. 100 auctions/sec should be ending. Mitigation: Standby acquires leadership via Redis lease in 2-4s. Backlogged auctions mass-closed in parallel by shard. Worst case: 12-second delay, during which late bids are accepted.
Clock Skew Causes Premature Closure
Redis clock drifts 5s ahead → auctions close early → legal liability. Mitigation: Clock skew guard in every Lua script (reject if caller and Redis TIME differ > 5s). NTP monitoring with 100ms drift alert threshold. Affected auctions can be reopened with time extension.
Shill Bidding Detected After Settlement
Buyer overpaid $4K due to artificial inflation. Payment captured, seller paid. Mitigation: Freeze seller account. Replay legitimate bids to compute fair price. Partial refund from seller's payout reserve. Real-time fraud detection prevents most cases pre-close.
WebSocket Tier Complete Failure
2M+ connections drop. No real-time updates. Mitigation: Clients fall back to HTTP polling every 3 seconds. Bid submission (REST) is unaffected. WebSocket reconnection with exponential backoff + jitter. STATE_SYNC message on reconnect catches up missed updates.
09
Interview Tips
Interview Tips
"The core challenge is that bids on the same auction must be serialized. Let me show why, and how that constraint shapes everything." This demonstrates you understand the fundamental challenge, not just the component list.
"500K closures/day × 8 bids = 4M/day ÷ 86K = ~50/sec average. But the peak on a single auction is what drives my architecture — 50 bids/sec on one key." The derivation matters more than the number.
Most candidates cover bidding + closure. Few discuss the Vickrey-like mathematical resolution, composite sort keys for tie-breaking, or information leakage prevention. This level of detail sets you apart.
Don't wait to be asked. "This creates a hot-key problem — I can't shard within one auction. Here's my mitigation: bid coalescing, hot auction isolation to dedicated shards, and update throttling for watchers."
"Strongly consistent on the write path via single-threaded Redis Lua, eventually consistent on the read path via replicas and pub/sub. This split is the fundamental decision — everything else follows."
"There's an entire state machine for settlement — payment holds, escrow, disputes, chargebacks. The most interesting challenge is chargebacks after seller payout — mitigated by reserve accounts and real-time fraud scoring."
10
Evolution
Evolution
How this design grows from MVP to planet-scale.
MVP — Single Server (0-10K Auctions)
PostgreSQL handles everything. SELECT FOR UPDATE for bid serialization. Direct WebSocket pushes from the app server. Simple setInterval scheduler. No Kafka — synchronous writes and notifications. Works fine at ~0.3 bids/sec total.
Growing Pains — Introduce Redis + Kafka (10K-1M Auctions)
Redis Lua replaces PG row locks for bid processing. Kafka added for async event processing. Read replicas for browse traffic. Separate WebSocket tier with Redis Pub/Sub. Elasticsearch for search. Basic fraud detection. Single Redis sorted set scheduler.
Scale-Out — Shard + Isolate (1M-10M Auctions)
Redis Cluster with 10-50 shards. Hot auction detection and migration to dedicated shards. Three-layer scheduler (PG → Redis sorted set → timing wheel). Bid coalescing. Update throttling. ML fraud detection. Multi-region reads, single-region writes.
Global Scale — Multi-Region Active-Active Reads (10M-100M)
Auctions assigned to nearest region for write serialization. Global read replicas with <50ms latency. Per-region schedulers. CDN-backed event streaming for viral auctions. Region-specific payment processors. Cross-region fraud aggregation.
Platform Evolution — Beyond Auctions (100M+)
Multi-format: auctions, fixed-price, reverse auctions, Dutch auctions. AI pricing recommendations. Dynamic reserves. Automated listing via computer vision. Integration marketplace for third-party tools. The auction engine becomes one module in a commerce platform.
Watch and read
References & Videos
Try next
Free to read · better with Enzo
Whiteboard this with Enzo
Enzo runs it as a live system design round on the whiteboard and grades your trade-offs.