The Garden of Forking Paths

Imagine a researcher who tests exactly one hypothesis, on one dataset, and reports the one result they got, with no p-hacking and no obvious cherry-picking. Surely that result is clean? Not necessarily — and the reason is a subtler problem than the usual "multiple testing" story about running twenty backtests and reporting only the winner.

The idea

The term comes from a Jorge Luis Borges short story about a garden where every path forks into more paths, all existing at once. Gelman and Loken borrowed the image for statistics: at almost every step of building an analysis, a researcher faces small forks — should returns be winsorized at the 1st or 5th percentile? Should the universe include micro-caps? Should the lookback window be 60 or 90 days? A researcher who is responding to the data as they go — checking whether each choice "looks right" before moving to the next step — ends up walking down one specific path through the garden, one that was implicitly selected because it produced a sensible-looking result at each fork. No single decision was dishonest. But the path as a whole was chosen, unconsciously, to work.

This is different from classic p-hacking, where someone runs many explicit tests and picks the best one. Here there might be only one analysis ever run — but it was shaped, fork by fork, by a stream of small, reasonable-looking judgment calls, each nudged by a glance at how the data was behaving. The researcher never "tried" the other paths, so there's nothing to disclose as a multiple-comparisons problem in the traditional sense, yet the effective number of implicit comparisons can be just as large as if they had.

A concrete example

A researcher is testing whether a sentiment signal predicts next-day returns. They decide to exclude stocks under $500m market cap "because microcaps are noisy" — a reasonable call, made after noticing the raw result was messier with them included. They then choose a 20-day rather than 10-day smoothing window, because 20 days "looked cleaner" in a quick plot. They settle on excluding earnings-announcement days, because a few extreme days were skewing the average. Each choice sounds defensible in isolation, and the researcher can honestly say they only ran the analysis once. But the final reported Sharpe ratio reflects a path through dozens of such forks, each one nudged toward significance by what the data showed at that step — the same effect multiple testing corrections exist to control for, just with the comparisons happening implicitly, one decision at a time, instead of as a batch of formal alternative tests.

What this means in practice

The fix isn't to stop making judgment calls — every analysis requires them. It's to make the forks before looking at how each choice affects the headline result: decide the universe filter, the window length, and the exclusion rules up front, ideally in a written research brief, rather than tuning them while watching the outcome. Where that's not possible, a sealed holdout sample lets a garden-of-forking-paths analysis be checked against data the researcher never touched while making those calls — if the result survives there, the specific path taken through the garden mattered less than the underlying signal.