Lesson 1 of 6 · 46 min

The ML system design framework

The spine every ML system design round rides — framing, success metrics, data, model, serving, monitoring — how the 60 minutes are actually budgeted, the six dimensions interviewers grade, and the three cardinal sins that silently sink strong engineers by minute 10.

What this round actually tests

The ML system design round is not a modeling quiz. It tests one thing: can you take a vague business goal and ride it down a fixed spine — framing → metrics → data → model → serving → monitoring — under a 60-minute clock, naming the tradeoff at every box. HelloInterview calls the field “the wild west,” where consistency among companies and interviewers is “frustratingly low,” which is exactly why the framework beats memorising any one company’s answer. This lesson is the spine the rest of the track hangs on, and the part most candidates rush.

Every credible framework collapses to the same nine stages. Chip Huyen’s Designing Machine Learning Systems lays them out over its first ten chapters; alirezadir’s widely-used template packages them as an explicit 9-step list: problem formulation, offline+online metrics, architectural components, data collection/labeling, feature engineering, model design/validation, prediction service, online testing/deployment, and scaling/monitoring. The structural insight to internalise: the spine is uncontroversial. Interviewers are not waiting for you to discover it — they are watching whether you can walk it without skipping framing, without starting at the model, and without going silent on failure.

System Design for Recommendations and SearchEugene Yan / MLOps.community

The spine: six beats, nine stages

Think of the round in six beats. (1) Frame — is this even an ML problem, what is the target function, who is the user, what is the scale and the latency budget. (2) Metrics — offline and online, each tied to a number and a counter-metric. (3) Data & labels — where the label comes from, its delay and noise, cold-start. (4) Model — baseline first, then candidate-gen → ranker → reranker, with two alternatives named. (5) Serving — the request path with a latency budget per stage, batch vs online vs streaming, caching. (6) Monitoring — drift signals, retraining cadence, the canary and rollback plan. Miss a beat and the interviewer has already pencilled in a gap.

code

1THE SPINE -- ride it top to bottom, name a tradeoff at each box23  FRAME      "is this ML? what's the target Y? user? scale? latency?"4     |          -> the wrong Y here poisons every later decision5  METRICS    offline (AUC/NDCG/Recall@K) + online (CTR/$/retention) + COUNTER6     |          -> the offline/online gap is the #1 production failure surface7  DATA       label source + delay + noise + sampling + leakage + cold-start8     |9  MODEL      baseline (popularity + logistic) -> candidate-gen -> rank -> rerank10     |          -> name 2-3 alternatives; defend the SIMPLEST that hits the metric11  SERVING    request path; batch vs online vs streaming; latency budget PER STAGE12     |13  MONITOR    drift signals + retrain cadence + canary 1-5% + rollback play1415  Senior tell: you draw this BEFORE picking a model, and you put a16  number on every arrow.

Interview angle. alirezadir’s own guidance is that seniority shows up as knowing which steps to abbreviate, not as covering all nine exhaustively. HelloInterview formalises this as a breadth/depth dial by level: roughly 80/20 for mid-level, 60/40 for senior, 40/60 for Staff+. A Staff candidate who “breezes through caches” to spend 60% of the time on the hard ranking tradeoff is signalling correctly; a mid-level candidate who drowns the interviewer in trivia and never finishes the design is not.

Beat 1 — framing: lock the target function in the first 10 minutes

The single most common way to fail is to skip framing and design for the wrong Y. Meta’s MLE prep is explicit: before proposing any model, map the prompt to a known ML objective — binary classification, learning-to-rank, edge prediction, retrieval-conditioned generation. State the supervised objective out loud in 30 seconds or you start on the back foot. The decision rule for “is this even ML?”: if the ML signal beats the next-best heuristic by more than ~3× on an offline metric, ML wins; otherwise push back and propose the rule. A popularity baseline is not a weakness — it is the thing you must beat.

Framing also means volunteering the numbers the prompt left silent. Real interview prompts plant them: a TikTok fraud round opens with “millions of transactions daily, end-to-end path under 300ms, model inference under 20ms, AUC-ROC and precision@K.” A ByteByteGo news-feed prompt confirms “~3B total, ~2B daily active, 2 refreshes/day, rank within 200ms.” When the prompt is silent on latency budget, scale, freshness, or label delay, say the number you are assuming and confirm it — that single move separates an MLE candidate from an SWE one.

Beat 2 — metrics: offline, online, and a counter-metric

Metrics split by task family. Classification → Precision, Recall, F1, PR-AUC, ROC-AUC; retrieval/ranking → mAP, NDCG, MRR, Recall@K; regression → MSE/MAE/RMSE. Online metrics are the business ones — CTR, conversion, revenue, dwell time, retention. The senior differentiator, marked as such by HelloInterview coaches and codified in alirezadir’s template, is that every metric needs a counter-metric: diversity for recsys, ad load for feeds, seller health for marketplaces, false-decline rate for fraud. Naming a counter-metric unprompted is one of the cleanest senior tells in the room.

The deepest point about metrics is the offline/online gap — the largest failure surface in production ML. Your offline AUC can climb while CTR falls, because you optimised the wrong construct. YouTube’s canonical example: they optimise expected watch time via weighted logistic regression, not click probability, precisely because clicks reward clickbait. Stripe Radar ties its threshold to a break-even precision derived from chargeback economics, not to a raw F1. The strong answer always weighs the metric back to a business unit — dollars per 1M impressions, chargeback dollars blocked — not log-loss.

The six dimensions interviewers actually grade

Across Exponent’s rubric, the Backprop FAANG guide, Meta MLE prep, and Jingfei Du (who has run “hundreds of ML System Design interviews”), grading collapses to six buckets. Weak candidates silently lose points in the first two — Problem Exploration and Data/Labels — because the interviewer has already downgraded them by minute 10. Breadth of coverage across all six is the single biggest predictor of a hire; the rule of thumb is never spend more than ~10 minutes in any one bucket.

code

1THE SIX-BUCKET RUBRIC (what's in the interviewer's head)23  1 Problem exploration   business goal, scope, scale, latency, success metric4  2 Data / label strategy label SOURCE + delay + noise + cold-start + sampling5  3 Feature engineering   user / item / context / cross; embeddings for unstructured6  4 Model architecture    retrieve-then-rank; multi-task vs N heads; justify w/ tradeoff7  5 Evaluation strategy   offline + online A/B + feedback loop; metric -> business $8  6 Serving & operations  per-stage latency budget, feature store, retrain, monitor910  Weak loses points in 1 & 2 (by minute 10).11  Staff+ is PENALIZED for missing nuance, not for skipping caches.

Interview angle. The buckets are not weighted equally and the weighting shifts with the prompt. For a recsys prompt, buckets 3–4 (features + the two-stage funnel) carry the most signal; for fraud, buckets 2 and 5 (delayed labels + cost-aware threshold) dominate; for an LLM/RAG prompt, a fourth axis appears — evaluation without ground-truth labels — which has the weakest industrial conventions and is where candidates most often improvise badly. Knowing which bucket a given prompt is really testing is itself a senior signal.

Budgeting the 60 minutes (and the 45-minute variant)

Sessions run 35–60 minutes, with 45 the most common target. Jingfei Du’s core observation: “many candidates fail to complete their design within the typical 45-minute to 1-hour timeframe, leaving interviewers without enough information.” The fix is a pre-computed budget you spend deliberately, checking in after each phase — “does this match what you’re looking for, or should I drill in?” — and explicitly saying “let me move on” when a section runs long. Finishing the whole spine beats a perfect ranking layer and no monitoring story.

code

160-MINUTE BUDGET (compress proportionally for 45)23  PHASE              MIN     LOCK THIS                         SKIP THIS4  ---------------    ----    ------------------------------    ------------------5  Clarify + frame    0-10    target Y, metric, latency, scale  business essays6  Metric design      10-15   offline + online + counter        "measure7                                                                 engagement"8  Architecture       15-30   end-to-end request flow, labeled  naming every tech9  Data + features    30-40   label source, hot/cold, embeds    every table schema10  Model design       40-50   baseline -> CG -> rank -> rerank  loss derivations11  Serve + monitor    50-58   per-stage latency, canary, drift  k8s manifests12  Wrap-up            58-60   the 3 tradeoffs you'd revisit      re-explain diagram

Three signals land at the senior bar, every time. You name a baseline by minute 15 (popularity + logistic regression). You express offline accuracy in business units, not log-loss. And you volunteer a counter-metric. Production-paper literacy is increasingly a tiebreaker — citing Covington’s YouTube DNN, Stripe Radar’s break-even precision, or Eugene Yan’s online/offline 2×2 in conversation signals you have shipped systems, not just studied them.

The three cardinal sins (and fast recoveries)

01Sin 1 — skipped clarifying questions, so you designed for the wrong Y. Recovery: restate the assumed target function explicitly and offer to re-prioritise the rest of the agenda.
02Sin 2 — started with the model (“we’ll use a transformer”). Recovery: draw the end-to-end architecture diagram first, then slot the model into the box where it belongs.
03Sin 3 — no online metric named, so you can’t show ownership of the funnel. Recovery: state the A/B metric on the spot — CTR, NDCG@10, chargeback-$ rate, faithfulness — plus the minimum detectable effect you’d target.
04Bonus tell — designing only the happy path. Recovery: pre-empt the silent failures (drift, label delay, cold-start, hot shards) before the interviewer asks.

In close calls, communication and proactive framing routinely outvote raw technical depth. — the consistent lesson from graded interviewing.io debriefs: the 4/4-communication candidate gets the offer the 4/4-technical rambler misses.

Interview prep

These are the framing-and-process questions that open almost every ML system design round, independent of the specific prompt. Answer each in 60–90 seconds, leading with the principle, then the concrete move.

01“How do you approach an ML system design question?” → frame the target Y + metric + scale + latency first; draw the end-to-end spine; defend each box with a number and a tradeoff; finish.
02“What’s the first thing you do?” → clarify scope and lock the supervised objective and success metric — not pick a model.
03“Is this even an ML problem?” → only if the ML signal beats the best heuristic by ~3× on an offline metric; otherwise ship the rule and revisit.
04“What metric would you optimise?” → an offline metric tied to an online business metric, plus a counter-metric (diversity / ad-load / false-decline).
05“How do you budget 45 minutes?” → ~10 clarify, ~5 metrics, ~15 architecture, ~10 data/features, ~10 model, ~5 serve+monitor; check in after each phase.
06“What separates a strong from a weak answer?” → breadth across the six buckets, a counter-metric, business-unit metrics, and unprompted failure modes — not depth in one place.
07“What would you skip if short on time?” → exact table schemas, loss-function derivations, k8s manifests — never framing, metrics, or the monitoring story.
08“How do you go from offline to online?” → ship to 1–5% via canary, watch guardrail metrics for an MDE, ramp over days; never “just deploy it.”

Going deeper, the follow-ups that probe seniority: “you’re running low on time, what now?” (say so, name the remaining tradeoffs you’d explore, and land the wrap-up); “why a baseline first?” (it sets the bar the fancy model must beat, and it ships in a day); “the interviewer keeps changing the requirements” (the Top-K class of prompts deliberately does this — show you adapt the data structure rather than defending your first sketch); and “what would you monitor?” (the answer is graded as heavily as the modeling — “watch accuracy on a hold-out” is the wrong answer; name drift signals, the business KPI, and the rollback play).

repoml-system-design.md — the 9-step templatealirezadir / Machine-Learning-Interviews docsMachine Learning System Design in a HurryHelloInterview docsDesigning Machine Learning Systems (the reference text)Chip Huyen articleSystem Design for Recommendations and Search (the online/offline 2×2)Eugene Yan

Checkpoint

An interviewer says “design a system to recommend products.” You have 45 minutes. What is the strongest first move?

ASketch a two-tower retrieval model and a DCN ranker, then explain the embeddingsBAsk for the catalog size and QPS so you can pick an ANN indexCClarify the business goal, lock the target function and success metric (plus a counter-metric), and confirm scale and latency — before drawing anythingDPropose a popularity baseline and start coding it

Checkpoint

Your ranker lifts offline NDCG@10 by 4 points. The interviewer asks, “how do you know this helps the business?” Which response is the WEAKEST (the one to avoid)?

AClaim that NDCG@10 is the gold-standard ranking metric, so a 4-point lift is a clear win on its ownBShip to 1–5% behind a canary and measure the online metric against an MDE over ~2 weeksCWatch a counter-metric like diversity alongside the primary online metric during the A/BDRun the new ranker in shadow first to catch latency or distribution regressions before exposing users

Checkpoint

You are interviewing at the Staff+ level. With 45 minutes, how should you allocate depth versus breadth compared to a mid-level candidate?

ASame as mid-level — cover every stage of the spine equallyBSpend more time on the hardest tradeoff (≈40/60 breadth/depth), brushing past commodity pieces like caches and regional replicationCSkip framing entirely since a Staff engineer is expected to know the objectiveDMaximise breadth and avoid going deep, to demonstrate range

Checkpoint

Halfway through, you’ve spent 20 of 45 minutes on feature engineering and haven’t touched serving or monitoring. What’s the right recovery?

AKeep going on features — depth there shows expertiseBQuietly skip serving and hope the interviewer doesn’t noticeCRestart the design with a tighter scopeDSay “let me move on to serving and monitoring to keep us end-to-end,” summarise features in one line, and budget the remaining time deliberately

Checkpoint

The prompt is “detect spam in comments.” Volume is low and a keyword blocklist already catches most of it. What’s the strongest senior response?

ADesign a fine-tuned transformer classifier immediately — spam detection is a classic ML taskBNote that ML is only justified if it beats the keyword baseline by a meaningful margin on an offline metric; propose shipping the rule now and layering ML on the residual tailCRefuse the problem because rules are sufficientDUse an unsupervised anomaly detector since labels are scarce

Could you take a cold prompt, frame it, walk the full spine, and field the process questions above in a 45-minute round?

Not yetGetting thereConfident

Takeaways

Every ML SD round rides one spine: frame → metrics → data → model → serving → monitoring. Draw it before picking a model.
Framing is decided in the first 10 minutes; the wrong target Y poisons everything downstream.
Every metric needs a counter-metric, and offline numbers must weigh back to a business unit — the offline/online gap is the #1 failure surface.
Six buckets are graded; weak candidates lose points in Problem Exploration and Data/Labels by minute 10.
Budget the clock, check in after each phase, and finish the spine — completion beats a perfect single section.
The three sins: skipped framing, model-first, no online metric. Pre-empt failure modes unprompted.

Next: data, features & embedding pipelines — where the label comes from, feature stores, and the leakage that fakes a great offline score.

Sources

Free to read · better with Enzo

Learn it with Enzo

Save your progress, answer the checkpoints, and let Enzo quiz you on what you just read.

Continue with Enzo Sign up free