Bad Controls and Post-Treatment Bias

A quant estimates the effect of a new execution algorithm on portfolio return, and — wanting to be thorough — controls for post-trade slippage in the regression, since slippage seems like an obviously relevant variable. The coefficient on the algorithm nearly vanishes. Did the algorithm really do nothing? No: slippage is caused by the algorithm choice itself — it's one of the channels through which the algorithm affects returns. Controlling for it blocks off exactly the mechanism you were trying to measure, mechanically forcing the estimated effect toward zero. This is a bad control, and it's one of the most common and hardest-to-notice mistakes in applied regression work.

An analogy: measuring a drug by controlling for its own symptoms relief

Suppose you test whether a painkiller reduces reported pain, but you also control for "hours of sleep last night" — and the drug's whole mechanism is that it lets people sleep better, which is what reduces their pain. Controlling for sleep blocks the very pathway you're studying; you'd conclude the drug "has no effect on pain, holding sleep constant," which is true but useless, since the drug never works any other way. The mistake isn't a statistical error — the regression runs fine and the standard errors are correct — it's a conceptual one: you controlled for a consequence of treatment as if it were a cause.

The distinction, one symbol at a time

Let $D$ be treatment (execution algorithm), $Y$ the final outcome (return), and $M$ a mediator — a variable that lies on the causal path from $D$ to $Y$, so $D \to M \to Y$. A bad control is any variable, like $M$, that is itself an effect of $D$. Including it in a regression of $Y$ on $D$ estimates:

$$\text{(direct effect of } D \text{ on } Y \text{, holding } M \text{ fixed)}$$

rather than the total effect

$$\text{(total effect)} = \text{(direct effect)} + \text{(effect of } D \text{ on } Y \text{ through } M).$$

In plain English: controlling for a mediator answers "what's left of the effect after we surgically remove the channel through which it operates?" — a different and usually less useful question than "what's the actual total effect of doing $D$?" The backdoor criterion (see confounding and the backdoor criterion) only calls for controlling variables that are not descendants of treatment; a mediator, by definition, is a descendant.

Worked example 1: the execution algorithm, decomposed

Suppose the algorithm's true total effect on return is +50 basis points, all of which flows through reduced slippage: it reduces slippage by 30bps, and every 1bp of slippage reduction translates to roughly 1.67bp of extra return (through better fill prices compounding across trades) — the arithmetic is illustrative, not universal. A regression of return on algorithm-dummy alone recovers close to +50bps. A regression that also controls for realized slippage effectively asks "what's the algorithm's effect on return if slippage hadn't changed" — and since slippage is exactly how the algorithm works, that residual direct effect is close to zero. Reporting the second regression's near-zero coefficient as "the algorithm's effect" badly understates its real, total impact.

Worked example 2: a subtler case with a collider twist

A firm studies whether a training program ($D$) improves trader retention ($Y$), and controls for "6-month performance review score" ($M$), reasoning that performance seems like a natural control. But performance review score is partly caused by the training itself ($D \to M$) and partly by an unmeasured trait like resilience, which also independently affects retention ($U \to Y$, with $U \to M$ as well). Controlling for $M$ here is doubly bad: it blocks part of the training's true effect (mediation) and, because $M$ is also a collider of $D$ and $U$, it opens a spurious path linking training to the unmeasured trait $U$ (see colliders and collider bias), contaminating the estimate with $U$'s effect on retention. Two distinct biases, from adding one seemingly reasonable control.

What this means in practice

The fix is not statistical but conceptual: before adding any control to a regression, draw (even mentally) whether it plausibly sits upstream of treatment (a legitimate confounder to control for) or downstream of treatment (a mediator or collider, generally a bad control). If you want the total effect of a policy, don't control for outcomes the policy itself produces. If you specifically want to decompose an effect into direct and indirect channels, formal mediation methods exist for that purpose (see mediation analysis) — but that's a deliberate, careful exercise, not something to stumble into by tossing every "relevant-looking" variable into a regression.