Regularization Bias in Causal Estimation

Ridge, lasso, and tree-based models are all built to trade a small amount of bias for a large reduction in variance — deliberately shrinking coefficients or pruning splits, because for pure prediction that trade almost always pays off. Plug one of these regularized models into a causal-effect formula, though, and the bias it was deliberately given for prediction purposes doesn't just make the effect estimate slightly noisy — it introduces a systematic distortion in the effect itself, one that doesn't shrink away even as the sample grows, because regularization bias is a property of the model class, not of sample size.

The idea: shrinkage bias transfers, and doesn't average out

Consider estimating a treatment effect using regularized regression to control for confounders $x$: fit a lasso model of the outcome on treatment and controls, and read off the treatment's coefficient. The lasso penalty shrinks coefficients toward zero to avoid overfitting the (possibly high-dimensional) set of controls — a sound choice for the controls, whose individual values aren't of interest, but this shrinkage doesn't stay contained to the control coefficients. Formally, the bias term in a regularized estimate of a target parameter $\theta$ looks roughly like

$$\text{bias}(\hat{\theta}) \approx \mathbb{E}\Big[\big(\hat{\eta}(x) - \eta_0(x)\big) \times (\text{something correlated with treatment})\Big]$$

In plain English: if the nuisance model $\hat\eta$ (here, the regularized control-variable fit) is systematically off from the truth $\eta_0$ in a way that's correlated with treatment assignment — which regularization bias typically is, since which controls get shrunk depends on the data, and treatment assignment is often related to those same controls — that correlated error doesn't average out over more data; it becomes a persistent bias in $\hat\theta$ itself. A plain regularized regression's estimating equation has no built-in mechanism to cancel this out, unlike a Neyman-orthogonal one.

Worked example: an omitted confounder shrunk away

Suppose the true relationship is that a trading algorithm's execution quality (the outcome) depends on order size (treatment-adjacent) and on a genuine confounder, market volatility at the time of the order, which also happens to influence which algorithm gets selected in the first place. A lasso model fit to predict execution quality from size, volatility, and a dozen other loosely related controls may shrink volatility's coefficient toward zero if its marginal contribution to prediction accuracy is smaller than a handful of other controls — reasonable for minimizing prediction error, since volatility might correlate with several other retained features. But if volatility is the true confounder linking order size to execution quality, shrinking its coefficient leaves some of its confounding influence unremoved, and the estimated size-execution effect absorbs part of what should have been attributed to volatility — a bias that persists no matter how much more data is collected, because it's driven by the model's regularization choice, not by sample noise that averages out.

What this means in practice

This is precisely the motivation for double machine learning and other Neyman-orthogonal approaches: instead of reading a treatment effect directly off a single regularized regression's coefficient, they construct an estimating equation whose sensitivity to nuisance-model error (including regularization bias) is zero to first order, cross-fit to avoid overfitting contamination on top of that. The practical takeaway is a warning, not a prohibition — regularized ML models are fine and often necessary as nuisance-model components in causal estimation, but only inside an orthogonalized, cross-fitted framework, never plugged directly into a naive difference-in-predictions formula.