A tiny p-value does not mean a bigger edge

Question

Two signals are tested. Signal A returns $p = 0.001$ ; signal B returns $p = 0.03$ . A colleague concludes, "A's edge is much bigger and much stronger than B's, since its p-value is 30 times smaller."

Explain why a smaller p-value does not imply a larger effect, and give a concrete counterexample where B actually has the bigger edge despite the larger p-value.

Accepted Answer

Why the ranking is unfounded. The p-value depends on three things braided together: the size of the effect, the amount of data, and the noise. Roughly, the z-statistic behind the p-value is z = \frac{	ext{effect size}}{	ext{standard error}} = \frac{	ext{effect}}{	ext{noise}/\sqrt{n}}. A small p-value can come from a big effect, from a huge sample, or from low noise, in any combination. So p-values cannot be used to rank effect magnitudes across studies that differ in sample size or noise. A concrete counterexample. Suppose: - Signal A: estimated edge = 1.0 bp, standard error = 0.30 bp, so…

A tiny p-value does not mean a bigger edge

Your answer

Discussion

Learn the concepts

Related questions