Portfolio-Level Versus Signal-Level Significance

A research team screens fifty candidate signals and finds that six of them clear the usual $t > 1.96$ bar. Is that six genuine discoveries, or is it about the number you would expect from fifty coin flips at a 5% false-positive rate? Separately, if the team instead combines all fifty signals into one portfolio and tests that portfolio's return, is the resulting single $t$-statistic held to the same standard as the individual ones? These are two different questions, and the confusion between them is one of the most common ways weak research gets waved through.

Two different hypotheses

Testing fifty signals individually means running fifty separate hypothesis tests, each asking "is this one signal's mean return zero?" Run enough of those and some will clear the bar by chance alone — screening at a 5% significance level across fifty independent, genuinely worthless signals should produce roughly two or three false positives on average, not zero. Picking the best-looking few after the fact and reporting only those is the multiple-testing problem in its purest form.

Testing the combined portfolio is a different, single hypothesis: "is the mean return of this one number — the equal- or risk-weighted blend of all fifty signals — zero?" There is exactly one test being run, so the ordinary 1.96 threshold applies honestly, with no correction needed. This is why quant desks often care less about any one signal's individual significance and more about whether the aggregate book clears a bar, provided the aggregate was specified in advance and not built by cherry-picking after seeing which signals worked.

Worked example

Fifty candidate signals are backtested. Assume, for illustration, that none of them has any real edge — every one is pure noise. At a 5% significance level you should expect about $50 \times 0.05 = 2.5$ of them to show $t > 1.96$ purely by chance. In a real screen, six clear that bar: three false positives more than expected under pure noise, roughly consistent with normal sampling variation around 2.5.

The corrected bar. A Bonferroni correction for fifty simultaneous tests divides the significance level by fifty, $0.05/50 = 0.001$, which corresponds to a critical $t$ near 3.5, not 1.96. Of the original six "significant" signals, suppose only one clears $t = 3.5$. The honest conclusion is that the screen has produced one plausible candidate out of fifty, not six.

The portfolio route. Now combine all fifty signals into a single equal-weighted composite and backtest that one portfolio. Because many of the fifty carry small pieces of a shared underlying signal, their combination's Sharpe ratio and $t$-statistic can legitimately be higher than any individual signal's — say $t = 4.8$ on the blended portfolio. This is a single test, run once, at the ordinary 1.96 threshold, and it passes cleanly. Two very different-looking numbers — a single surviving signal at $t=3.5$ under a corrected bar, and a combined portfolio at $t=4.8$ under no correction — can both be honest reads of the same fifty-signal screen.

	What is being tested	Number of tests	Threshold
Signal-level screen	Each signal's own mean return	50	Corrected, e.g. Bonferroni $t \approx 3.5$
Portfolio-level	One combined portfolio's mean return	1	Ordinary $t \approx 1.96$

In practice

• Pre-register the aggregation rule. A portfolio-level test is only a single honest test if the combination weights were fixed before looking at which signals performed well. Choosing weights after seeing the results smuggles the multiple-testing problem back in.
• Report how many signals were tried, not just how many worked. A single significant result out of 50 attempts and a single significant result out of 3 attempts are not the same finding, even with an identical $t$-statistic.
• Combining weak, roughly independent signals into one portfolio is not cheating — it is exactly what a diversified aggregate is supposed to do, and it is why The Fundamental Law of Active Management rewards breadth. The danger is specifically in testing many candidates and reporting only the survivors as if they were the only ones tried.
• The The Deflated Sharpe Ratio formalises the correction for exactly this situation: it raises the bar for a Sharpe ratio's significance as a function of how many strategy variants were tried before the reported one was chosen.
• Correlation between the fifty signals changes the correction too. Fifty highly correlated signals are not fifty independent tests; see Estimating Breadth With Correlated Bets for how to count the effective number of tests when the candidates overlap.