Splitting your alpha across five interim looks

Question

You plan to run an A/B test but want to check the results at five evenly spaced points during the experiment, stopping early if a look is convincing. You still want the overall chance of a false positive across all five looks to stay at 5%.

Using a simple Bonferroni split, what significance threshold should you require at each look?

Accepted Answer

The problem with five free looks. Each time you test at 0.05, a truly null experiment has a 5% chance of falsely crossing. Do that five times and the chances of at least one false crossing pile up well above 5%. To keep the overall error at 5%, each individual look must use a stricter threshold. Bonferroni split. The simplest safe rule divides the total error budget equally among the looks: \alpha{	ext{per look}} = \frac{\alpha{	ext{overall}}}{	ext{number of looks}} = \frac{0.05}{5} = 0.01. So require p < 0.01 at each of the five looks. Because the chance of any single false crossing is no…

Splitting your alpha across five interim looks

Your answer

Discussion

Learn the concepts

Related questions