Data

Compute the lift, SE and CI: ship or not?

Asked at

Airbnb

1Give yourself 5 minutes

2Answer out loud, not in your head

3Then compare with the answer below

Reference answer

Lift = (0.056 - 0.05)/0.05 = +12% relative. SE_pooled = sqrt(p_pool(1-p_pool)(1/n_t + 1/n_c)) where p_pool = (0.056120k + 0.05120k)/240k = 0.053. z = 0.006 / SE_pooled ~ 3.7, p ~ 0.0002. CI: 0.006 +/- 1.96 * SE ~ (0.003, 0.009) absolute -> (6%, 18%) relative. Statistical significance real. Practical significance: depends on guardrails, latency, downstream margin. Ship if guardrails fine and the lower bound on lift exceeds the cost of the change - flag the CI as wide because MDE was set wider; consider extending for a tighter CI.

Then expect these follow-ups

Heterogeneous effects; what if SE is computed incorrectly (delta method) - what's the natural log alternative?

Free to read · better with Enzo

Practice this out loud with Enzo

Enzo runs it as a mock interview, pushes back with follow-ups, and grades you on the rubric.

Continue with Enzo Sign up free

Next question