Checking a retention test every morning for a month

Question

A mobile game studio tests a new onboarding tutorial, measuring day-7 retention. The experiment is meant to run for 30 days. The product manager checks the dashboard every morning and has told the team: "the first day it shows a significant win, we lock it in." On day 11 the tutorial is ahead with $p = 0.045$ .

Is a day-11 significant result trustworthy under this habit, and what is the sound way to run the test?

Accepted Answer

The daily habit is the problem. A p-value threshold of 0.05 controls errors at 5% only for a single, pre-planned look. Looking every morning and stopping at the first significant day is optional stopping: each day is a fresh opportunity for random noise to push the result across the line. Over roughly a dozen looks, the chance that a genuinely useless tutorial crosses 0.05 at least once is far above 5%, commonly in the 15% to 25% range, and it keeps climbing the longer you keep looking. So a day-11 crossing under this rule carries much weaker evidence than a plain reading of "p = 0.045" sugge…

Checking a retention test every morning for a month

Your answer

Discussion

Learn the concepts

Related questions