One midpoint check plus a final look

Question

You want to run an A/B test but with a single interim check halfway through (to catch a clear early winner or a harmful variant), plus the final analysis at the end, two looks in total. You want the overall false-positive rate across both looks to stay at 5%.

Using a simple Bonferroni split, what threshold should each look use?

Accepted Answer

Why even two looks need adjusting. Testing at 0.05 twice gives noise two chances to cross, pushing the combined false-positive rate above 5%. Each look needs a slightly stricter threshold. Bonferroni split. Divide the budget equally between the two looks: \alpha{	ext{per look}} = \frac{0.05}{2} = 0.025. So require p < 0.025 at both the interim and final looks. The total chance of a false crossing is then at most 2 	imes 0.025 = 0.05. Perspective. With only two looks the penalty is mild: 0.025 instead of 0.05. This is why adding one well-chosen interim check (for example, a safety look to ki…

One midpoint check plus a final look

Your answer

More Statistics questions