Ten weekly peeks and the price of each one

Question

You plan to run an A/B test for ten weeks and want to look at the results at the end of each week, ten looks in total, stopping early if the evidence is strong. You need the overall false-positive rate across all ten looks to stay at 5%.

Using a simple Bonferroni split, what threshold should each weekly look require?

Accepted Answer

The issue with ten looks. Ten checks at 0.05 give noise ten chances to cross the line; the combined false-positive rate would be far above 5%. Each look must use a much stricter threshold. Bonferroni split. Divide the total error equally: \alpha{	ext{per look}} = \frac{0.05}{10} = 0.005. So require p < 0.005 each week. The total chance of any false crossing is then at most 10 	imes 0.005 = 0.05. What this reveals. More looks mean a stricter bar at each one: with ten looks you need p < 0.005, five times stricter than the usual 0.05. That is the real cost of frequent peeking, each early look …

Ten weekly peeks and the price of each one

Your answer

Discussion

Learn the concepts

Related questions