Pair Selection and Multiple Testing

Take 500 stocks, form every possible pair, and test each one for cointegration at the usual 5% significance level. There are roughly 125,000 pairs. Even if not a single one is genuinely cointegrated, about 5% of them — over 6,000 pairs — will pass the test purely by chance. Picking the "best" ones from a screen like that isn't finding real relationships; it's cherry-picking noise.

Why the naive approach fails

A single cointegration test controls the chance of a false positive for that one test. It says nothing about the chance of at least one false positive across a whole universe of tests, and that chance grows fast: testing $n$ independent pairs at significance level $\alpha$, the probability of at least one spurious pass is roughly $1 - (1-\alpha)^n$, which is already over 99% once $n$ exceeds a few hundred. A desk that screens thousands of pairs and trades the twenty with the lowest p-values is, without correction, mostly trading statistical noise dressed up as an edge.

Worked example

A desk screens 20,000 pairs and finds 1,050 pass a cointegration test at the standard 5% level. At 5% significance with no real signal at all, chance alone predicts about $20{,}000 \times 0.05 = 1{,}000$ false positives — meaning the 1,050 "hits" are statistically indistinguishable from pure noise. Applying a Bonferroni correction — dividing the significance threshold by the number of tests, $0.05 / 20{,}000 = 0.0000025$ — and re-checking which pairs still clear that far stricter bar leaves perhaps a handful of genuinely surprising results, the ones actually worth investigating economically.

What this means in practice

The practical fix is two-layered: apply a multiple-testing correction (Bonferroni is conservative but simple; a false-discovery-rate procedure is less punishing and more common in practice) to decide which statistical results survive scrutiny, and separately require an economic story — a real shared business link — for any pair before trading it, since statistics alone cannot distinguish a lucky pair from a true one.