A significant signal, but is it real?

Question

Your signal-screening test uses significance level $\alpha = 0.05$ (its type I / false-positive rate) and power $1 - \beta = 0.80$ (so its type II rate is $\beta = 0.20$ ). From experience, only about $10\%$ of the candidate signals you screen have any real edge.

When a signal comes back "significant," what is the probability it truly has an edge?

Accepted Answer

Why the errors alone are not enough. \alpha and \beta tell you how the test behaves given the truth. But you want the reverse: given a positive result, how likely is a real edge? That flip requires the base rate (prior), so this is a Bayes' rule problem. Imagine screening 1000 candidate signals. Count the outcomes. - Real edges: 10\% of 1000 = 100. The test catches them with power 0.80, giving 100 	imes 0.80 = 80 true positives. - No edge: the other 900. The test falsely flags them at rate \alpha = 0.05, giving 900 	imes 0.05 = 45 false positives. Combine. Total positives = 80 + 45 = 125, o…

A significant signal, but is it real?

Your answer

More Statistics questions