HardData

Investigate a 15% CTR drop

Asked at

Netflix

1Give yourself 5 minutes

2Answer out loud, not in your head

3Then compare with the answer below

Reference answer

First, instrument: did the model roll out cleanly to 100%? Did server latency shift? Second, segmentation: where is it 15%? Vertical, country, browser, time-of-day? Third, ablations: did the click position change (position bias)? Did the model prefer new content (novelty) or did personalization break (skew in feature embedding)? Fourth, ranking diagnostics: side-by-side NDCG on a recent batch; check if rankings are still differentiating or if a tie-breaking rule now dominates. Decide: ship rollback vs side-by-side test.

Then expect these follow-ups

How would you ship a fix without breaking trust in the experimentation platform?

Free to read · better with Enzo

Practice this out loud with Enzo

Enzo runs it as a mock interview, pushes back with follow-ups, and grades you on the rubric.

Continue with Enzo Sign up free

Next question