HardData

Investigate a 15% CTR drop

Asked at

Netflix

1Give yourself 5 minutes

2Answer out loud, not in your head

3Then compare with the answer below

Reference answer

First, instrument: did the model roll out cleanly to 100%? Did server latency shift? Second, segmentation: where is it 15%? Vertical, country, browser, time-of-day? Third, ablations: did the click position change (position bias)? Did the model prefer new content (novelty) or did personalization break (skew in feature embedding)? Fourth, ranking diagnostics: side-by-side NDCG on a recent batch; check if rankings are still differentiating or if a tie-breaking rule now dominates. Decide: ship rollback vs side-by-side test.

Then expect these follow-ups

How would you ship a fix without breaking trust in the experimentation platform?

Hiring now

Get the Netflix interview, not just the prep

landed applies to roles like these for you, on each company’s own site, with answers written for the role.

Get 5x interviews10 free auto-applies. No card.

Or practice this question first

Next question