Cross-Fitting and Sample Splitting

Neyman orthogonality protects a causal estimate from small, unbiased errors in the nuisance ML models it depends on — but it says nothing about overfitting. If the same data is used both to fit a flexible outcome model and to evaluate the treatment-effect formula that model feeds into, the model's predictions will fit that data's noise unusually well, and the resulting effect estimate inherits a bias from that in-sample overfitting that orthogonality alone cannot fix. Cross-fitting is the practical companion to orthogonality: split the data so the nuisance models are never evaluated on the same rows they were trained on.

The idea: fit on one half, evaluate on the other, then swap

Split the sample into $K$ folds (commonly $K=2$ or $K=5$). For each fold $k$, fit the nuisance models (outcome model, propensity model) using all data except fold $k$, then use those out-of-fold models to compute the causal-estimate contribution for the rows in fold $k$ only. After cycling through all $K$ folds this way, average the per-fold estimates into the final answer:

$$\hat{\theta} = \frac{1}{K} \sum_{k=1}^{K} \hat{\theta}_k, \qquad \hat{\theta}_k \text{ computed using } \hat{\eta}^{(-k)} \text{ on fold } k$$

In plain English: $\hat{\eta}^{(-k)}$ denotes the nuisance models trained on every fold except $k$; the effect contribution for fold $k$'s rows is always computed using a model that never saw those specific rows during training, exactly the same discipline as evaluating a predictive model out-of-sample. Averaging across all $K$ ways of leaving one fold out uses every row exactly once for evaluation while still using nearly the whole dataset for training each nuisance model.

Worked example: two-fold cross-fitting on a small causal dataset

A dataset of 1,000 trades is split into fold A (500 trades) and fold B (500 trades). Round 1: fit the outcome and propensity models on fold A only, then use those models to compute each fold-B trade's contribution to the treatment-effect estimate — fold B's rows were never touched during fitting, so there's no overfitting contamination for those 500 estimates. Round 2: fit fresh outcome and propensity models on fold B only, and use them to compute each fold-A trade's contribution. The final estimate averages all 1,000 per-trade contributions, each one computed from a nuisance model that never saw that trade during training. Compare this with the naive alternative — fitting the outcome model on all 1,000 trades and evaluating the formula on those same 1,000 trades — where a sufficiently flexible model (say, a deep gradient-boosted tree) can fit in-sample noise well enough to make the treatment effect look artificially precise or shifted, an error that doesn't shrink as the sample grows because it's driven by overfitting, not by ordinary sampling variability.

What this means in practice

Cross-fitting is the standard companion to Neyman-orthogonal estimators like the DR-learner and double machine learning, and is what allows a practitioner to plug in genuinely flexible, high-variance ML models (random forests, boosted trees, neural nets) as nuisance estimators without worrying that the models' overfitting will leak into and bias the causal effect they're used to compute. The cost is a modest efficiency loss (each nuisance model is trained on less than the full dataset per fold) and roughly $K\times$ the computation, since a full set of nuisance models must be trained per fold — a trade almost always worth making given the alternative of a silently biased causal estimate.