Event Count Versus Observation Count in Signal Tests
Why the number of independent events behind a signal test — not the number of raw rows in the dataset — is what actually determines how much you should trust the result, and how the two numbers can differ by orders of magnitude.
Prerequisites: Effective Sample Size With Autocorrelated Observations, Hypothesis Testing
A researcher backtests an earnings-announcement signal on 10 years of daily data across 500 stocks — 1.25 million rows. That number looks huge, and a p-value computed as if it came from 1.25 million independent observations will look extremely convincing. But the signal only actually fires around four times a year per stock, on earnings days, and stocks in the same sector tend to react to related news together. The real number of independent pieces of evidence behind the test might be a few hundred distinct earnings events, not 1.25 million rows — and a test that doesn't recognize this difference will report false confidence that the signal is real.
Two very different counts
The observation count is just how many rows are in your dataset — every stock-day, every trade, every tick. The event count is the number of genuinely distinct, roughly independent occurrences the signal's logic actually depends on. These coincide only when every row is an independent draw of new information, which is rare in finance: daily returns overlap information across days (autocorrelation), stocks in the same industry share news (cross-sectional correlation), and a strategy that only trades around specific triggers (earnings, mergers, index rebalances) only really produces a new "event" each time that trigger occurs, no matter how many stock-days you stack around it.
Standard errors and p-values computed with the plain observation count treat every row as independent evidence, which makes them far too small (and t-statistics far too large) whenever the true event count is much smaller. This is the same problem clustered standard errors are designed to fix — the fix here is a matter of degree: you have to be honest about which axis (time, sector, single-event-window) the correlation is riding on, and shrink your effective sample size accordingly before trusting a significance test.
A rough way to estimate it
A simple, practical estimate: count the number of genuinely distinct trigger events (earnings releases, not stock-days around them; distinct merger announcements, not every day of the deal). If the signal fires 4 times a year per stock across 500 stocks over 10 years, that's roughly 20,000 potential events — still large, but two orders of magnitude smaller than 1.25 million, and the honest number to plug into a significance calculation. If many of those events cluster on the same handful of calendar dates (all Q4 tech earnings landing in the same week, say), the effective count shrinks further still, because those events aren't fully independent of each other either.
The number of rows in a dataset is not the same as the number of independent events supporting a signal's test. Overlapping time periods, correlated cross-sectional groups, and infrequent trigger events all shrink the true event count far below the observation count — and using the raw row count in a significance test overstates confidence, sometimes drastically.
"We tested this on over a million data points" is a common but misleading way to describe statistical power. If those million points reduce to a few hundred genuinely independent events once autocorrelation and clustering are accounted for, the true statistical power is that of a few-hundred-observation study, not a million-observation one — and the reported significance should be recalculated on that basis before anyone trusts it.
Related concepts
Practice in interviews
Further reading
- Harvey, Liu & Zhu, 'and the Cross-Section of Expected Returns', Review of Financial Studies (2016)