Infrastructure
Design LeetCode
01
What the judge must do
What the judge must do
- A user picks a problem and a language, pastes source code and submits it for judging against hidden test data.
- The result is one verdict: Accepted, Wrong Answer, Time Limit Exceeded, Memory Limit Exceeded, Runtime Error or Compilation Error, plus runtime and peak memory.
- A separate "try it" action runs the code on the visible examples or on custom input and shows stdout, with no effect on stats.
- Timed contests with a live scoreboard ranked by solved count, then by penalty time.
- Each problem has its own time and memory limits, and limits can be scaled per language (Python gets a multiplier).
- Users can browse their submission history and reopen any old source file.
- Untrusted code must never reach the host, the network or another user's files.
- Verdicts are reproducible: the same code on the same tests gives the same verdict and similar timing, run after run.
- During contests, p95 time from submit to verdict stays under 8 seconds.
- No submission is lost once the API has returned an ID, even if a runner dies mid-execution.
- Hidden test data is never exposed through output, errors or timing side channels that leak a whole file.
Ask how big the largest contest is and how quickly people submit after it opens, because that burst sizes the whole runner fleet. Ask whether interactive problems (where the program talks to a grader) and special checkers (problems with many correct outputs) are in scope. Finally, ask whether timing must be comparable across submissions. If yes, you need dedicated CPU cores per run, which changes the cost model more than anything else on this page.
02
Sizing the fleet
Sizing the fleet
Assume 6 million submissions on a normal day, two thirds of them "try it" runs on examples and one third full judgements. A weekly contest draws 60,000 people over 90 minutes. Each full judgement runs about 80 test files on average.
| Quantity | Working | Result |
|---|---|---|
| Average submission rate | 6M / 86,400 s | about 70 / s |
| Full judgements per day | 6M x 1/3 | 2M |
| Contest submissions | 60K users x 7 attempts on average | about 420K in 90 min |
| Contest peak rate | final 10 minutes carry about 20% of attempts | about 140 / s |
| CPU per full judgement | compile 1.5 s + 80 tests x 25 ms average | about 3.5 core-seconds |
| Cores busy at contest peak | 140 / s x 3.5 core-s, plus normal traffic | about 700 cores |
| Runner hosts at peak | 700 / 28 usable cores, 30% headroom | about 33 hosts |
| Source storage per year | 6M x 365 x 2.5 KB, compressed 4:1 | about 1.4 TB |
| Test data | 4,000 problems x 50 MB average | about 200 GB, cached on every host |
| Scoreboard updates | one per accepted verdict | under 50 / s |
Request volume is tiny by web standards. The cost is CPU seconds, and the shape of the load: a flat baseline with a cliff every Sunday. The fleet you need for the last ten minutes of a contest sits mostly idle the rest of the week, which is why autoscaling and a pre-warmed pool matter more than any database choice.
200 GB of inputs and expected outputs fits on one NVMe drive. Sync it to every runner host and read it locally instead of fetching from object storage for each submission.
03
API design
API design
Submitting returns 202 immediately and the verdict arrives over a WebSocket or by polling. Holding an HTTP request open for the duration of compilation and 90 test runs would tie up gateway connections exactly when the contest burst hits.
04
Architecture overview
Architecture overview
flowchart LR
U["Browser"] --> G["API gateway"]
G --> SA["Submission API"]
SA --> DB[("Postgres: submissions")]
SA --> BS[("Object store: source files")]
SA --> Q[["Priority queue: contest, judge, scratch"]]
Q --> D["Runner agent on host"]
D --> SB["Sandbox: compile then run"]
TD[("Local test data cache")] --> SB
SB --> D
D --> V[["Verdict topic"]]
V --> SA
V --> SC["Scoreboard service"]
SC --> RZ[("Redis sorted sets")]
V --> WS["WebSocket push"]The submission API is the only piece the browser talks to. It checks the language, size and rate limits, writes a row with state queued, puts the source file in object storage and pushes a small job record onto the queue. The job holds the IDs, the language, the problem version and the limits. It never holds the test data, and the source is fetched by the runner rather than packed into the message.
Runner hosts are where the risk lives, so they are kept boring. Each host runs one trusted agent that pulls jobs, prepares a sandbox, feeds it inputs from a local read-only copy of the test data, compares output and reports back. User code only ever exists inside the sandbox. The agent publishes a verdict event, and three consumers react: the submission API updates the row, the scoreboard service adjusts standings for contest jobs, and a push service forwards the result to the open WebSocket.
Contest jobs, full practice judgements and scratch runs live in separate lanes. Agents always drain the contest lane first and take scratch runs only when the others are empty, so the Sunday burst never waits behind someone testing a print statement.
A trusted daemon per host that owns a pool of idle sandboxes, pins each run to dedicated cores, enforces limits, and acknowledges the queue message only after the verdict is published. If it dies, the message reappears for another host.
Every problem version is a content-addressed bundle synced from object storage to each host's disk. Jobs reference the bundle hash, so a test fix made mid-contest cannot mix old and new files inside one judgement.
One Redis sorted set per contest with a packed score of solved count and penalty. Ranks are read with ZREVRANK, and pages with a range query, so standings stay fast at 60,000 rows.
05
Deep dive: running hostile code fast and fairly
Deep dive: running hostile code fast and fairly
You are running thousands of untrusted programs a minute and you owe each one an exact verdict. That needs two things that fight each other: strong isolation, which costs startup time, and fast turnaround, which punishes every millisecond of setup. This section is about getting both.
Layers of isolation
No single mechanism is enough, so stack them. The outer layer is the boundary between user code and the host kernel. Plain containers share that kernel, and one kernel bug turns into a host compromise. A user-space kernel such as gVisor intercepts system calls and answers most of them itself, which shrinks the exposed surface for a modest syscall overhead. A microVM such as Firecracker gives each run its own guest kernel, the strongest wall, with a boot in the low hundreds of milliseconds and a few megabytes of memory overhead.
Inside that wall, apply the usual Linux controls. A cgroup caps memory (the limit plus a small allowance for the runtime), process count (blocks fork bombs) and CPU. A seccomp profile allows only the syscalls a compiled program or interpreter needs. Network namespaces are empty, so there is simply no interface to talk to. The filesystem is a read-only image of the compiler toolchain plus a small tmpfs scratch directory capped at, say, 64 MB. The process runs as an unprivileged user with every capability dropped.
Finally, the agent watches from outside: a wall-clock timer kills the run at twice the time limit even if CPU time is low, which catches programs that sleep or block on input forever.
flowchart TD
J["Job pulled from contest lane"] --> W["Take a warm sandbox for cpp20"]
W --> C{"Compiles within 10 s?"}
C -->|no| CE["Verdict: compilation error"]
C -->|yes| T["Run next test with cgroup limits"]
T --> K{"Exit, time and memory OK?"}
K -->|no| BAD["Verdict: TLE, MLE or runtime error"]
K -->|yes| CMP{"Output matches checker?"}
CMP -->|no| WA["Verdict: wrong answer on test i"]
CMP -->|yes| MORE{"More tests?"}
MORE -->|yes| T
MORE -->|no| AC["Verdict: accepted"]
CE --> R["Destroy sandbox, refill pool"]
BAD --> R
WA --> R
AC --> RWarm pools hide the startup cost
If each job had to boot a microVM, mount an image and start a JVM, setup alone could exceed the run itself. So each agent keeps a pool of sandboxes already booted for the popular languages, sized from recent demand: perhaps eight C++ sandboxes, six Python, four Java, two of everything else. A job claims one, runs, and the sandbox is destroyed afterwards, never reused. Reuse would let one user leave a file or a background process for the next. The agent boots a replacement in the background. With Firecracker snapshots you can restore a pre-initialised VM in tens of milliseconds, which makes the pool cheap to refill.
Compilation happens inside the same sandbox as execution, with its own limits (10 seconds, 1 GB). Compilers have had bugs that hang or blow up on crafted input, and a template-heavy C++ file can be a denial of service on its own.
Measuring time so verdicts are fair
Contestants compare their runtimes, and a Time Limit verdict on a solution that passed yesterday is a support ticket. Three habits keep timing honest. Pin each run to dedicated physical cores with hyperthread siblings left idle, so a neighbour cannot steal cache or execution units. Measure CPU time from the cgroup, not wall-clock time, and use wall-clock only as the safety kill. And disable turbo frequency scaling on runner hosts, because a chip that boosts when cool and throttles when hot makes the same program vary by 20 percent.
Per-language multipliers are a policy choice, not physics. A common approach sets the limit for the reference C++ solution, then grants interpreted languages a fixed multiplier, and checks during problem setup that a reasonable Python solution passes.
Checking output
Most problems compare tokens after trimming whitespace. Floating point problems compare with a tolerance like 1e-6. Problems with many valid answers ship a checker program, written by the problem setter, that reads the input, the user output and the reference answer and returns a verdict. The checker runs in a sandbox too, with its own limits, because user output can be huge or malformed. Interactive problems swap the static input file for an interactor process connected to the user program by two pipes, both inside the same isolated environment.
For contests, stop at the first failing test. That saves compute and returns verdicts faster when the cluster is busiest. Practice judgements can run everything and report how many passed, which helps learners.
"Each run gets a fresh, never-reused sandbox: a microVM or gVisor boundary, then cgroups, seccomp, no network and a read-only filesystem. The agent keeps a warm pool per language to hide startup, pins each run to its own cores, and reports CPU time from the cgroup. Contest jobs drain first and fail fast. Test data is content-addressed and local, so every judgement uses exactly one version."
06
Mistakes to avoid
Mistakes to avoid
A program can leave a background process, a file in /tmp or a modified environment behind. The next user inherits it, and their verdict, or their source code, is no longer private.
A noisy neighbour or a garbage collector on the host adds hundreds of milliseconds, so correct solutions get Time Limit on busy days and contestants lose rating points.
Messages become megabytes, the broker slows during the burst, and test files end up copied into logs and dead letter queues where anyone with broker access can read them.
With a few dozen deliberate failures, a user can print the hidden inputs one by one and hardcode the answers.
Sixty thousand contestants refresh standings every few seconds near the end, and a GROUP BY over every submission locks up the primary.
07
Decisions and what they cost
Decisions and what they cost
A kernel-level wall between user code and the host, at the price of some boot time and memory per sandbox. The warm pool turns that boot time into background work, and snapshots make refills quick.
Isolation you can defend in a security reviewFastest to start and cheapest to run. But every run shares the host kernel, so one privilege escalation bug exposes the whole host, including other users' code and the test data. Acceptable for an internal tool, not for public traffic.
Too thin for untrusted codeCuts CPU use by roughly half on wrong submissions and returns the verdict sooner. Contestants only need to know that they failed and roughly where.
Contest lane default"Passed 71 of 92" tells a learner whether the idea is close or wrong. It costs more compute, which is fine on the practice lane where nobody is racing a clock.
Practice onlyStable, comparable timing. The cost is lower packing density: you leave hyperthread siblings idle and run fewer jobs per host than the raw core count suggests.
Fairness is the productCounting instructions with hardware counters is perfectly deterministic and lets you pack hosts densely. It also ignores cache misses and memory bandwidth, so it can reward code that would be slow on real hardware. Some judges use it; most still prefer CPU time.
Worth mentioningA durable queue absorbs the burst and lanes protect contest latency. Runner hosts scale up ahead of each scheduled contest because start times are known days in advance.
Predictable bursts, planned capacityAttractive for spiky load, but most function platforms limit syscalls, cold starts are unpredictable for heavy runtimes, and you lose control over core pinning, which wrecks timing fairness.
Timing becomes noise08
Failure modes
Failure modes
Someone submits code that probes a kernel exploit. With plain containers, success means root on a runner that also caches every problem's hidden answers.
Keep a kernel boundary (microVM or gVisor) between code and host, run hosts with no credentials beyond reading test bundles, rebuild hosts from images regularly, and alert on any denied syscall pattern that looks like probing.Submissions arrive faster than runners finish. Verdicts that normally take 3 seconds take 90, and contestants resubmit, which makes the backlog worse.
Scale runner hosts up an hour before the start, pause the scratch lane during the contest, cap in-flight jobs per user, and record queue time separately so penalty is based on submit time, not verdict time.A solution near the time limit passes on one host and fails on another. Contestants post side by side screenshots and trust in the ranking drops.
Run identical hardware in the contest pool, pin cores, disable turbo, and automatically re-run any Time Limit verdict within 10 percent of the limit on a second host before publishing it.A program spawns processes in a loop or prints gigabytes to stdout. Without limits it starves the host, and the agent tries to buffer all that output in memory.
A pids limit in the cgroup stops the fork bomb, and output is read through a capped pipe: past the limit, the run ends with an Output Limit verdict.A setter discovers that test 33 has a wrong expected answer. Hundreds of people have been marked wrong.
Publish a fixed bundle under a new hash, then rejudge every affected submission against it from a dedicated rejudge lane, and recompute standings from the new verdicts.A solution leaks to a chat group and dozens of accounts submit near-identical code within minutes.
Run similarity checks on normalised token streams or syntax trees after the contest, flag clusters for human review, and hold rating changes until review ends.09
How to handle the interview
How to handle the interview
Lead with the threat model. Say out loud that every request is hostile code and list what it might try: loop forever, exhaust memory, fork, open sockets, read the test files. Then map each threat to a control.
Make submission asynchronous. Return an ID straight away and push the verdict later. Interviewers want to see you separate the API from the slow, risky work.
Size the fleet from CPU seconds, not requests. Seventy requests a second sounds small until you multiply by compile and test time and apply the contest burst.
Explain how you keep timing fair. Dedicated cores, CPU time from cgroups and a re-run near the limit show you understand that the verdict is the product.
Name the priority lanes. Contest, practice and scratch runs have different stakes. Draw them, and explain what you throttle when the cluster is full.
Version the test data. Content-addressed bundles make rejudging clean and stop a mid-contest fix from mixing files inside one run.
Treat the scoreboard as its own small system. A sorted set updated by verdict events, plus a freeze in the final hour if the product wants suspense.
Mention what you return on failure. Only the test index for hidden tests, never the input, or you have built a test data download service.
10
Growing the design in stages
Growing the design in stages
A web app writes jobs to a Postgres table and a worker process on the same machine runs each one in a locked-down container. Good enough for a university course, as long as nobody hostile shows up.
Runners move to their own machines with no access to the main database. A message queue replaces the job table, and verdicts flow back as events.
Contests bring motivated attackers and rating disputes. Add microVM or gVisor sandboxes, priority lanes, core pinning and versioned test bundles.
Startup cost dominates, so agents keep warm sandboxes per language restored from snapshots. Capacity scales up ahead of each scheduled contest, and the single shared worker pool is gone.
Runner pools in several regions cut latency for scratch runs, while contest judging stays on one identical hardware pool for fairness. Plagiarism detection and automatic rejudging become standard tooling.
Watch and read
Try next
Free to read · better with landed
Whiteboard this with landed
landed runs it as a live system design round on the whiteboard and grades your trade-offs.