Sampling Bias and Selection Effects

Every statistic you compute describes the sample you actually have, not the population you wanted to describe — and those two are only the same thing if the sample was collected fairly. Sampling bias is what happens when the process that put data in front of you was not neutral, so your "average" answer is really the average of whoever, or whatever, happened to make it into the dataset.

An analogy: the gym survey

A gym wants to know how satisfied its members are, so it hands out a survey at 7am to whoever is currently working out. The people who show up at 7am and stick with the gym long enough to be there that morning are, almost by definition, the ones who are happiest with it — the dissatisfied members already quit and are nowhere in the room to be surveyed. The gym will get a glowing satisfaction score that says nothing true about the average person who ever signed up. The bias isn't in the answers — it's in who got asked in the first place.

The mechanism, precisely

Let $Y$ be the outcome you care about (a fund's return, a stock's performance, a survey answer) and let $S$ be an indicator — 1 if that observation is included in your sample, 0 if not. The quantity you actually compute is the sample average conditional on inclusion:

$$E[Y \mid S = 1] \ne E[Y]$$

Read this as: "the average of $Y$ among the observations you kept" is not, in general, equal to "the true average of $Y$ across everyone." The gap between the two sides is the size of the bias, and it exists whenever whether an observation gets included ($S$) is correlated with the outcome itself ($Y$) — funds that performed badly close down and stop reporting, so $S=0$ (excluded) is correlated with $Y$ being low.

Where it hides in finance data

Survivorship bias: a database of "all hedge funds" today only lists funds that are still operating. Funds that blew up or closed are gone, and they were disproportionately the bad performers — so the average return of "all funds" as measured today overstates the true historical average by several percentage points a year.

Look-ahead-adjacent selection: building a backtest universe from "the current S&P 500 constituents" and running it back 20 years silently excludes every company that was delisted, acquired, or dropped from the index for underperforming — the survivors, again, look better than the full original population did.

Self-selection: a broker's client survey on "how did our execution algo perform" only hears from clients who are still clients — the ones who left because execution was bad never answer.

The explorer above lets you draw repeated samples from a true population and watch the sample mean bounce around the truth — that bouncing is sampling error, which shrinks with more data.

The running-average explorer above shows that same shrinkage directly: the line settles onto the truth as $n$ grows, because each new draw comes from the whole population, honestly. Sampling bias is different and worse: imagine every "draw" instead came only from survivors — the running average would still settle down, just onto the wrong number, and no additional data would ever pull it back toward the truth.

Worked example 1: survivorship bias in a toy fund universe

Ten hedge funds launch with $100m each. Over five years, seven survive and return, on average, $8\%$ annualized. Three blow up and are removed from the database, having returned $-40\%$ before closing.

True average return across all ten funds: $\frac{7 \times 8 + 3 \times (-40)}{10} = \frac{56 - 120}{10} = \frac{-64}{10} = -6.4\%$.

Average return reported by a database that only lists today's surviving funds: simply $8\%$, because the three failures were dropped entirely, not averaged in at a low or negative number — they are absent, not counted as zero. The gap between the honest $-6.4\%$ and the survivorship-biased $8\%$ is $14.4$ percentage points a year, entirely an artifact of who was still around to be measured.

Worked example 2: a backtest universe built with hindsight

A backtest tests a "buy the biggest 50 companies" strategy using today's list of the 50 largest S&P 500 stocks, applied to prices going back 15 years. Suppose over that period, 8 of today's top 50 were small or mid-cap 15 years ago and grew into giants — precisely because they performed spectacularly — while several companies that were top-50 15 years ago (and later shrank or were delisted) are missing from today's list and thus missing from the backtest entirely.

If those 8 high-growth survivors averaged $18\%$ annualized over the period versus $9\%$ for the other 42, the backtest's reported return is $\frac{8 \times 18 + 42 \times 9}{50} = \frac{144 + 378}{50} = \frac{522}{50} = 10.44\%$. Rerun honestly using the actual top-50 list as it stood at each date in the past — which would include some names now delisted, say averaging $4\%$ instead of the vanished names' true contribution — and the honest number could easily be several points lower. The backtest never "lied" about any single number; it simply never saw the companies that would have dragged the average down.

What this means in practice

• Always ask "how did an observation get into this dataset, and could that process be correlated with the outcome I'm measuring?" before trusting a summary statistic built from it.
• Use point-in-time universes — the actual list of tradeable securities as of each historical date — rather than today's list projected backward, which is the direct fix for the backtest example above; see point-in-time data.
• Treat any "average performance" statistic from a public database (funds, strategies, traders) as an upper bound, not an estimate, unless the source explicitly states it includes closed or failed entities.