A per-look threshold for four interim analyses

Question

A clinical-style A/B test will be evaluated at four evenly spaced interim points, with the option to stop early. You want the overall false-positive rate across all four looks to remain 5%.

Using a simple Bonferroni split, what significance threshold should each look use?

Accepted Answer

Why a correction is needed. Testing at 0.05 at each of four looks gives noise four separate chances to cross, so the combined false-positive rate would exceed 5%. Each look must use a stricter threshold to keep the total in budget. Bonferroni split. Divide the total error equally among the looks: \alpha{	ext{per look}} = \frac{0.05}{4} = 0.0125. So require p < 0.0125 at each of the four looks. Then the total chance of any false crossing is at most 4 	imes 0.0125 = 0.05. Caveats. Bonferroni is a worst-case bound that ignores the correlation between successive looks (later data includes earli…

A per-look threshold for four interim analyses

Your answer

More Statistics questions