Lesson 4 of 6 · 46 min

Monitoring, drift & retraining

The monitoring story is graded as heavily as the modeling — “watch accuracy on a hold-out” is the wrong answer. Online vs offline metrics, the drift taxonomy and which statistical test to use, shadow vs canary, the three layers of rollback, the retraining loop, and a worked incident scenario.

The bucket candidates treat as an afterthought

The research is unambiguous: the monitoring story is graded as much as the modeling story, and “watch accuracy on a hold-out” is the canonical wrong answer — because in production you usually don’t have fresh labels, the distribution moves, and the failure shows up as a silent business-KPI regression, not a unit-test failure. This lesson is the operations layer: the metric set, the drift taxonomy and which test to fire, shadow vs canary, the three layers of rollback, the retraining loop, and the incident scenario interviewers love to spring.

Start by separating the two metric planes. Offline metrics (AUC, NDCG, recall@K on a held-out set) are what you compute before shipping. Online metrics (CTR, dwell time, chargeback-$ rate, faithfulness) are what you watch in production. The trap is monitoring only model-internal signals; you must also monitor the business KPI directly, because statistical drift tests catch distribution shifts but not causal regressions — a model can become subtly more biased on the tail or over-promote one merchant category while every input distribution looks unchanged. The senior framing: pair statistical drift monitors with the actual business metric and a shadow challenger.

How to Detect and Interpret Data Drift in ProductionEmeli Dral (Evidently AI) — MLconf NYC

The drift taxonomy: three different things called “drift”

Chip Huyen’s taxonomy, sharpened by Evidently, distinguishes three. Data / covariate drift: the input distribution P(X) shifts (a new device type, a new geography, a marketing campaign changing the traffic mix). Label / prior drift: the marginal P(Y) shifts (fraud rate jumps during a holiday). Concept drift: P(Y|X) changes — the relationship between inputs and the label moves, which is the dangerous one because the same input now means something different (fraudsters adopt a new tactic; user taste shifts). Naming which kind you suspect, and why it implies a different response, is the senior tell. Concept drift usually needs retraining on fresh labels; covariate drift may just need re-calibration or re-weighting.

code

1DRIFT DETECTION TESTS -- which to fire (Evidently empirical study)23  TEST                      STRENGTH                FAILURE MODE / USE4  -----------------------   ---------------------   --------------------------------5  Kolmogorov-Smirnov (KS)   most accurate           TOO SENSITIVE on big N (flags 1%6                                                      drift at n=100k); use for7                                                      n<1000 or when 1% matters8  Population Stability (PSI) standard for prod       misses subtle shifts; thresholds:9                                                      <0.1 stable, 0.1-0.2 minor,10                                                      >0.2 retrain11  Wasserstein distance      distribution-aware       costlier at scale; mid-tier sets1213  KEY: statistical drift != business regression. Always pair with the BUSINESS KPI14  (CTR, take-rate, refund rate) + a shadow challenger on a 1-5% slice.

Interview angle. “Which drift test would you use?” is a real probe, and the precise answer earns points. The Kolmogorov-Smirnov test is the most accurate but becomes too sensitive at large N — it flags a 1% drift at n=100,000 — so it’s right for samples under 1,000 or when even 1% matters. PSI is the production standard for “major changes,” with the canonical thresholds PSI <0.1 stable, 0.1–0.2 minor, >0.2 retrain. The deeper point, worth stating unprompted: a statistical test will never catch a silent business regression, so it’s a tripwire, not the verdict.

Shadow then canary: the safe rollout order

The recommended deployment order is shadow → canary → full rollout, and the distinction matters. A shadow deployment receives mirrored production traffic in parallel with the live model but its outputs are recorded, not returned — this is where you catch distribution shifts, latency variance, and latent bias with zero user risk. A canary routes a small slice (1–5%) of real user traffic to the new model and serves its output — this is where you catch user-visible regressions and KPI dips. Shadow validates safely but can’t catch user-impact; canary catches user-impact but exposes a slice; the order is shadow first to de-risk, then canary to confirm.

Never say “deploy the model.” The senior phrasing is concrete: “ship to 5% behind a canary, watch the guardrail dashboard against an MDE for a few days, ramp 5%→10%→50%→100% only if no business-KPI regression.” This connects back to the offline/online gap from Lesson 1 — the canary is how you discover that an offline AUC lift didn’t move (or hurt) the online metric, before it reaches everyone. A/B with a control vs test arm and significance testing is the measurement; bandits are the faster-adapting variant when you want to shift traffic toward the winner automatically.

Rollback has three layers — and the slowest is the feature store

When a deploy goes wrong, rollback is not one action — it’s three layers with very different speeds. Model version: swap the weights back (fast, with a blue-green pattern and a model registry). Config version: revert the threshold, prompt template, or routing rules (fast). Feature version: revert feature definitions in the store — and this is the slow, dangerous one, because the online store doesn’t atomically clear; stale values linger until they TTL out or you force-flush, which can poison predictions for minutes. GMI Cloud blames “No Model Versioning Strategy” — the absence of versioned files and blue-green patterns — for slow, chaotic rollbacks. The senior move: name all three layers and flag that feature rollback is the one with a tail.

The retraining loop and its cadence

Retraining cadence is a real design decision, not “retrain weekly.” The standard production loop: a drift alarm (or a fixed schedule) triggers, you freeze the feature definitions, retrain on a frozen set that includes the recent window, shadow the new model 24–48 hours, shadow-compare against the champion on the same slice, then canary 5% for 1 hour → 10% for 4 hours → 50% for 24 hours → full, gated by a human review on the business-KPI. The whole loop is typically 3–7 days. Cadence is driven by how fast the world moves: fraud retrains monthly (Stripe found one-month-newer data yields up to +0.5pp recall), while a stable churn model might retrain quarterly.

Cadence has a counterintuitive ceiling. In fast-label-arrival domains like fraud, the rate at which ground-truth labels mature actually limits how aggressively you should retrain — over-parameterized models retrained too fast will overfit to outdated patterns before the recent labels have matured. This is why Stripe Radar’s model pack is deliberately linear/logistic regression and gradient-boosted trees with a small neural net on top, not the latest transformer: the label dynamics, not model capability, set the complexity ceiling. Interview angle. “How often would you retrain?” → tie it to label maturation and drift rate, and note that faster isn’t always better.

code

1THE RETRAINING LOOP -- gated, typically 3-7 days end to end23  drift alarm OR schedule4        |5  freeze feature definitions  (so train and serve agree)6        |7  retrain on frozen set incl. recent window8        |9  SHADOW new model 24-48h  -> shadow-compare vs champion on same slice10        |11  CANARY  5% (1h) -> 10% (4h) -> 50% (24h) -> 100%12        |            gated by a HUMAN review on the business KPI at each step13  full rollout (or roll back across 3 layers)1415  Cadence by domain: fraud ~monthly (+0.5pp recall from newer data),16  stable churn ~quarterly. Faster != better -- labels must mature first.

A worked incident: the Saturday fraud spike

The canonical fraud incident, straight from the research: “the model looked good in offline eval but the chargeback rate spiked because a fraud cluster shipped on a Saturday when retraining hadn’t yet caught up.” Walk the response like an on-call engineer. Detect: the business-KPI monitor (daily chargeback-$ rate) fires before any statistical test, because it’s a concept shift the input distributions don’t reveal; a BIN-attack cluster (failed-card fingerprints in a 1-hour window) corroborates. Contain: tighten the rules-engine threshold for the affected vertical immediately — rules act in microseconds, while retraining takes days. Diagnose: shadow-compare the champion against a challenger trained on the last 24 hours. Recover: canary the challenger, then retrain on the matured labels once the chargebacks confirm.

Interview prep

Monitoring questions reward operational specificity. Lead with the business KPI, then the drift tripwire, then the rollout/rollback mechanism. Answer each in 60–90 seconds.

01“What would you monitor?” → the online business KPI directly + ML-specific drift signals (prediction/feature distribution, calibration, staleness) + latency/cost; never just hold-out accuracy.
02“Which drift test?” → PSI for production “major change” (>0.2 retrain); KS only for small samples or when 1% matters (it’s too sensitive at large N).
03“Data drift vs concept drift?” → P(X) shift vs P(Y|X) shift; concept drift needs fresh-label retraining, covariate drift may just need re-weighting/calibration.
04“Shadow vs canary?” → shadow mirrors traffic (outputs recorded, zero user risk, catches distribution/latency); canary serves a 1–5% real slice (catches user-visible regressions). Shadow first.
05“How do you deploy a new model?” → ship to 5% behind a canary, watch guardrails vs an MDE, ramp 5→10→50→100 only if no KPI regression.
06“How do you roll back?” → three layers — model (fast), config (fast), feature definitions (slow; the online store doesn’t clear atomically). Pin and version all three.
07“How often do you retrain?” → tie cadence to label maturation + drift rate; fraud ~monthly (+0.5pp recall from newer data); faster isn’t always better.
08“Chargeback rate spiked over the weekend — what now?” → business-KPI alert fires first; tighten rules immediately (microseconds), shadow-compare a challenger, then retrain on matured labels.

Going deeper, the follow-ups that probe maturity: “you have no fresh labels — how do you even know the model degraded?” (proxy signals: prediction-distribution drift, calibration drift, abrupt feature-distribution shifts, and the business KPI — labels are a lagging confirmation, not the trigger); “how do you detect a brand-new fraud pattern?” (concept-drift monitoring plus an unsupervised anomaly layer for the unlabeled tail); “the drift alarm fires every day — now what?” (it’s too sensitive — switch from KS to PSI, widen the window, or threshold; an alarm that always fires is ignored); and “how do you avoid a feedback loop?” (the model’s own outputs become tomorrow’s training data — log propensities, hold out an exploration slice, and watch diversity as a counter-metric).

articleWhich test is best? Comparing 5 methods to detect data driftEvidently AI articleShadow deployment vs. canary release of ML modelsJFrog ML (Qwak)articleA primer on machine learning for fraud detection (retraining cadence, +0.5pp recall)Stripe videoIntro to ML Monitoring: Data Drift, Quality, Bias and ExplainabilityWhyLabs

Checkpoint

An interviewer asks how you’d monitor a deployed ranking model in production. Which answer is strongest?

ATrack the online business KPI directly (e.g. engagement/revenue) plus ML drift signals — prediction- and feature-distribution drift, calibration, staleness — and run a shadow challenger; treat labels as lagging confirmationBCompute accuracy on a held-out test set every nightCAlert only when the model server’s error rate spikesDRe-run offline NDCG weekly and ship if it’s stable

Checkpoint

You set up a Kolmogorov-Smirnov drift test on a feature with ~200,000 daily samples. It fires a drift alarm almost every day. What’s going on and the fix?

AThe feature genuinely drifts daily; retrain every dayBKS is too sensitive at large N — it flags ~1% drift at n=100k+; switch to PSI (retrain only when >0.2) or widen the window / add a thresholdCYour sample size is too small; collect more dataDThe test is broken; remove drift monitoring

Checkpoint

You want to validate a new fraud model on real traffic distributions without risking a single wrong customer decision. Which rollout step fits?

ACanary it to 5% of live traffic immediatelyBFull rollout with a fast rollback readyCShadow deployment — mirror production traffic to the new model and record its outputs without returning them to usersDRun it offline on last month’s data again

Checkpoint

You roll back a bad model by swapping the weights to the previous version, but predictions stay degraded for several minutes. Most likely cause?

AThe load balancer is still routing to old podsBThe online feature store still serves the new model’s materialized feature values, which don’t revert atomically — they linger until they TTL out or you force-flushCThe model weights didn’t actually changeDGPU memory needs to be cleared

Checkpoint

For a fraud model, the interviewer asks how often you’d retrain. Which answer shows the most judgment?

AContinuously / hourly to stay maximally freshBTie cadence to label maturation and drift rate — roughly monthly for fraud (newer data buys ~+0.5pp recall), with a drift alarm able to trigger an off-cycle retrain; faster isn’t always betterCOnce a year, since fraud models are stableDOnly when accuracy on a hold-out set drops

Could you design a monitoring + drift + rollback + retraining story and walk an incident in a live round?

Not yetGetting thereConfident

Takeaways

Monitor the online business KPI directly plus label-free drift tripwires — “accuracy on a hold-out” is the canonical wrong answer.
Know the drift taxonomy: covariate P(X), prior P(Y), concept P(Y|X); concept drift is the dangerous one and needs fresh-label retraining.
PSI is the production drift standard (>0.2 retrain); KS is too sensitive at large N — use it only for small samples.
Roll out shadow → canary → full; roll back across three layers, and remember the online feature store doesn’t clear atomically.
Tie retraining cadence to label maturation and drift rate — fraud ~monthly for +0.5pp recall; faster overfits to noise.
Walk an incident like on-call: detect via the KPI, contain with rules (microseconds), diagnose with a shadow challenger, recover via canary + retrain.

Next: two fully worked classic-ML designs — recommendations and fraud — end to end with architecture, metrics, real numbers, and tradeoffs.

Sources

Free to read · better with Enzo

Learn it with Enzo

Save your progress, answer the checkpoints, and let Enzo quiz you on what you just read.

Continue with Enzo Sign up free