Testing Whether Two Signals Have Different IC

Signal A shows a sample IC of 0.04 over the backtest period; Signal B shows 0.03. Is A actually better, or is that gap just noise? The instinct is to reach for a standard two-sample test, treating the two IC estimates as independent — but they're usually not independent at all, since both signals were typically evaluated on the same stocks over the same dates. Ignoring that overlap and using an ordinary two-sample test overstates how surprising the observed gap is, making a difference look statistically significant when it's really within the range of noise you'd expect from two signals with genuinely identical true skill.

An analogy: comparing two students graded on the same exam

Suppose two students each guess answers on the same multiple-choice exam, and you want to know if one is genuinely better at guessing than the other. If you treated their scores as coming from two completely separate, independent exams, you'd miss an important fact: on any question where the correct answer happened to be an "obviously wrong-looking" option, both students might have been fooled together, and on any question with an "obviously right-looking" option, both might have gotten it right together. Their scores move together across questions, driven by shared question-level quirks, not just by each student's independent skill. Comparing them properly means looking at the question-by-question difference in their scores, not comparing the two overall averages as if they came from separate, unrelated exams.

The method, one symbol at a time

Because both signals' period-by-period ICs, $\text{IC}_A(t)$ and $\text{IC}_B(t)$, are measured using the same stocks and dates $t$, they share common shocks — a month where the whole market behaves unusually (a sharp reversal, a factor crash) tends to push both signals' ICs in the same direction together. This makes $\text{IC}_A(t)$ and $\text{IC}_B(t)$ correlated across time, and the right way to test whether their long-run averages differ is a paired test on the period-by-period differences, $d(t) = \text{IC}_A(t) - \text{IC}_B(t)$, rather than an unpaired test comparing the two average ICs as if drawn independently.

The paired t-statistic is

$$t = \frac{\bar{d}}{s_d / \sqrt{T}},$$

where $\bar{d}$ is the average of the period-by-period differences, $s_d$ is their standard deviation, and $T$ is the number of periods — computed entirely from the differences, so any common shock that moved both signals' ICs together in the same period cancels out of $d(t)$ and doesn't inflate the perceived noise in the comparison. This is the same logic as the Diebold-Mariano test used for comparing forecast accuracy: pair by time period first, then test the average difference.

Worked example: paired versus unpaired comparison

Suppose over 24 monthly periods, Signal A's IC averages 0.04 with a standard deviation across months of 0.10, and Signal B's IC averages 0.03 with the same standard deviation of 0.10. An unpaired (independent two-sample) test would compute a standard error for the difference in averages as $\sqrt{0.10^2/24 + 0.10^2/24} \approx 0.0289$, giving $t = (0.04-0.03)/0.0289 \approx 0.35$ — nowhere near significant. But suppose the two signals' monthly ICs are actually correlated at $\rho = 0.7$ across months (plausible, since they're both partly driven by the same market regime). The paired-difference standard deviation is instead $s_d = \sqrt{0.10^2 + 0.10^2 - 2 \times 0.7 \times 0.10 \times 0.10} \approx \sqrt{0.006} \approx 0.077$, giving a paired standard error of $0.077/\sqrt{24} \approx 0.0157$ and $t = 0.01/0.0157 \approx 0.64$ — still not significant here, but notice the paired standard error is nearly half the unpaired one; with a slightly larger true gap or a higher correlation between the signals, the paired test can flip a comparison from "not significant" to "significant" (or the reverse) purely by correctly accounting for the shared-shock structure that the unpaired test ignores entirely.

What this means in practice

This distinction matters whenever comparing a candidate replacement signal to an incumbent, or comparing two variants of the same signal family (different lookback windows, different universes), since these comparisons are almost always evaluated on overlapping stock-date panels and therefore share common shocks. Using an unpaired test in this setting tends to produce an overly conservative (too wide) standard error when the signals are positively correlated, potentially hiding a real, statistically meaningful improvement — or in cases of negative correlation between the signals, an unpaired test can understate the true uncertainty. Either way, the paired approach that differences out shared time-period effects first is the right default whenever two signals are being compared on the same underlying data.