Five looks, but with a stricter 1% overall budget

Question

A high-stakes experiment (a change that would be expensive to get wrong) will be checked at five evenly spaced interim points. Because the cost of a false positive is high, you want the overall false-positive rate across all five looks to be only 1%, not the usual 5%.

Using a simple Bonferroni split, what threshold should each look require?

Accepted Answer

Two things are stricter here. You have five looks (so noise gets five chances to cross), and you have chosen a tighter overall budget of 1%. Both push the per-look threshold down. Bonferroni split. Divide the chosen total error equally among the looks: \alpha{	ext{per look}} = \frac{\alpha{	ext{overall}}}{	ext{number of looks}} = \frac{0.01}{5} = 0.002. So require p < 0.002 at each of the five looks. The total chance of any false crossing is then at most 5 	imes 0.002 = 0.01. Interpretation. Setting the overall budget is a business choice: the more painful a false positive, the smaller yo…

Five looks, but with a stricter 1% overall budget

Your answer

Discussion

Learn the concepts

Related questions