A Library of Negative Results

Quant research teams generate far more failed ideas than successful ones, but most shops only keep records of what worked. Without a record of what was tried and rejected, researchers re-test the same signal variants every year or two as staff turn over, and — more seriously — nobody can tell how many hypotheses were quietly discarded before a "successful" one surfaced, which is exactly the information needed to judge whether that surviving signal is real or just the winner of a multiple-testing lottery.

A negative results library is simply a maintained log: every hypothesis tested, its specification, the data and period used, and the outcome, kept regardless of whether it passed or failed. Two things fall out of keeping it. First, it prevents duplicated effort — a quick search before starting new research shows whether a variant of the idea was already tried and why it failed. Second, and more important statistically, it lets a reviewer compute an honest multiple-testing correction: if the library shows 200 related hypotheses were tested before this one survived, the true significance of the survivor needs to be adjusted downward accordingly, something impossible to do if the 199 failures were never recorded.