Selection Bias in Trading and Execution Data

Trading and execution data is riddled with a specific kind of distortion: the data you get to see is not a random sample of the data that exists, because whether an observation shows up at all depends on the outcome you're trying to study. A limit order that never fills leaves no execution-quality record — only filled orders do — and fills are not random: they happen disproportionately when the market moved toward the order, meaning the price was, in some sense, already favorable. Studying "execution quality" using only filled limit orders therefore studies a biased subsample skewed toward favorable outcomes, and will make limit orders look better than they actually performed across every order that was placed, filled or not.

This is a close cousin of the classic sample-selection problem in econometrics, and it shows up throughout a trading operation. A backtest of "strategies currently running" ignores every strategy that was shut down for underperforming — a survivorship-style bias that flatters whatever process is still active. A model trained on "trades that passed compliance review" learns from a subsample where the riskiest-looking trades were already filtered out, understating tail risk in anything downstream. A study of "clients who kept trading after a fee increase" excludes exactly the clients most sensitive to fees, biasing any estimate of price sensitivity toward zero.

Worked example

A desk wants to measure the average slippage on limit orders placed one tick inside the spread. Using only orders that actually filled, average slippage looks like +0.3 ticks (i.e., favorable) relative to the price at order placement. But a limit order fills specifically when the market moves toward it — so the filled subsample overrepresents the exact scenario in which the order's price was retrospectively a good one. Pulling in the unfilled orders and asking what would have happened had they been sent as market orders instead (their next best alternative) shows an average cost of −0.6 ticks across that group. The true picture — averaging filled and the counterfactual cost of unfilled orders together — is close to break-even, nowhere near the rosy +0.3 the filled-only sample suggested. Reporting execution quality from fills alone overstates the strategy's edge by roughly a tick per order, purely from which orders happened to leave a record.

What this means in practice

The fix is almost always the same: define the population you actually care about before looking at outcomes — every order placed, every strategy ever launched, every trade ever submitted for review — and account for what happened to the ones that dropped out, rather than analyzing whatever subset happens to remain. This is more work than pulling a convenient "filled trades" table, but the alternative is an execution-quality, backtest, or risk estimate that's biased in a specific, predictable direction: too favorable, because the unfavorable cases were the ones most likely to disappear from the record.