Hypothesis-First vs Data-First Research

You have a dataset with four thousand columns and a laptop that will test all of them against next month's returns before lunch. Something will come back with a t-statistic above 3. The question that decides whether you have a career is not "did I find something?" — you always will — but "how surprised should I be that I found it?" That depends almost entirely on something the test statistic never sees: how many things you looked at before this one.

The DNA analogy

A crime lab reports a DNA match with a one-in-a-million chance of occurring by accident. How convinced should you be?

Entirely depends on how the suspect was found. If detectives already suspected him for reasons unconnected to the DNA — he was there, he had a motive, his story did not hold up — a one-in-a-million match is overwhelming. If instead the lab swept a database of ten million people and he was the one who came back, that coincidence is exactly what you should expect: about ten people in the database match by accident, and you have no idea which one you are holding.

Identical evidence, identical probability, opposite conclusions — because of how the candidate was selected. Hypothesis-first research is the detective with a suspect. Data-first research is the database sweep.

Counting the looks

Let $N$ be the number of distinct things you tested — variants, columns, parameter settings, universes, anything you tried and could have reported. Let $\alpha$ be the false-positive rate accepted on each test: the chance pure noise clears your bar, conventionally $\alpha = 0.05$, a 5% chance per test. If the tests are independent and no idea is real, the chance that at least one clears the bar is

$\Pr(\text{at least one false hit}) = 1 - (1-\alpha)^N.$

In English: the chance of being fooled at least once is one minus the chance of never being fooled, and the chance of never being fooled shrinks fast as you keep trying.

A second, more useful number. If you run $N$ tests on pure noise and report the best, the winner's t-statistic is not a typical draw — it is the largest of $N$ draws, far out in the tail. Its typical size is close to

$t_{\max} \approx \sqrt{2 \ln N},$

where $\ln$ is the natural logarithm. In English: the best of $N$ worthless ideas still looks about this impressive, so anything below the line is not evidence. Call it the noise ceiling.

Worked example 1: a 200-variant sweep

You try 200 sensible variants of a raw data field: four lookback windows, five smoothings, two neutralisation schemes, five universes. One comes back with a t-statistic of 3.0 and a backtest Sharpe Ratio of 0.95 over ten years. Is it real?

Step 1. Expected false hits: $N \times \alpha = 200 \times 0.05 = 10$. On pure noise, ten variants would look "significant".

Step 2. Chance of at least one false hit: $1 - 0.95^{200}$. Since $0.95^{200} = e^{200 \ln 0.95} = e^{-10.26} \approx 0.000035$, that is $0.999965$ — near certainty.

Step 3. The noise ceiling: $\sqrt{2 \ln 200} = \sqrt{2 \times 5.30} = \sqrt{10.60} = 3.26$.

Your winner scored 3.0; the ceiling for a 200-way sweep is 3.26. The best variant underperformed what noise alone would have produced. Not weak evidence — negative evidence. Note how the Sharpe hid this: a t-statistic is roughly Sharpe times the square root of years, so 0.95 over ten years gives $t \approx 0.95\sqrt{10} \approx 3.0$. A Sharpe near 1 over a decade sounds like a strategy right up until you count the looks.

Worked example 2: the same p-value, two beliefs

Take the same test — 5% false-positive rate, 80% power, meaning that if the effect is real the test finds it 80% of the time. Let $\pi$ be your prior probability that the idea is real before you ran anything. Bayes' rule gives

$\Pr(\text{real} \mid \text{it passed}) = \frac{\pi \times 0.80}{\pi \times 0.80 + (1-\pi) \times 0.05}.$

In English: of everything that passes, what fraction was genuinely real? Real ideas pass 80% of the time, fakes sneak through 5% of the time, and the answer is the first pile divided by both.

Hypothesis-first. You reasoned that index funds must buy S&P additions on the effective date, so there should be predictable pressure. One test, and a mechanism: call it a one-in-four shot, $\pi = 0.25$.

$\Pr = \frac{0.25 \times 0.80}{0.25 \times 0.80 + 0.75 \times 0.05} = \frac{0.200}{0.200 + 0.0375} = \frac{0.200}{0.2375} = 0.842.$

Data-first. You swept 200 columns with no story attached to any of them, and perhaps one of the 200 is real. So $\pi = 1/200 = 0.005$.

$\Pr = \frac{0.005 \times 0.80}{0.005 \times 0.80 + 0.995 \times 0.05} = \frac{0.0040}{0.0040 + 0.04975} = \frac{0.0040}{0.05375} = 0.074.$

Same data, same p-value, same code: 84% versus 7%. The thinking you did before opening the file is worth a factor of eleven in how much you should believe the output.

The sliders below make the point mechanically. Set a confident prior and feed in a little data: the posterior barely moves. Flatten the prior and the same data throws the answer around.

When data-first is the right call

None of this makes exploration wrong. It is the only sane approach on a genuinely new dataset where nobody has a prior worth having, when profiling data for breakage rather than alpha, and when generating hypotheses you will test somewhere else. The rule is not "don't look" — it is look on data you have agreed not to draw conclusions from.

The protocol is mechanical. Split the sample before you start and move the holdout somewhere you cannot reach (Holdout Discipline and the Sealed Sample). Explore freely on the rest, logging every variant tried, because the count $N$ feeds every correction you will need later and memory always undercounts. Then take your one or two survivors to the sealed data, once, and accept the number. A second visit and the holdout is training data.

What this means in practice

• Log the count. Every correction you might apply next — a The Deflated Sharpe Ratio, a Bonferroni cut, a false-discovery-rate control — needs $N$. A researcher who cannot say how many variants they ran cannot have their result evaluated by anyone, including themselves.
• Raise the bar with the search. Harvey, Liu and Zhu argue that after decades of published factor-hunting, a new equity anomaly needs a t-statistic near 3.0, not 2.0. That is the noise ceiling applied to a whole profession's search.
• Demand a mechanism. "Who is on the other side, and why do they keep trading with me?" A data-first hit with no answer should be sized as though it is probably noise, because the arithmetic above says it probably is.
• Grade ideas by prior. A mechanism plus one test is a strong claim; no mechanism plus a thousand tests is a lottery ticket. Knowing which you have changes how hard you defend it.