Beating the Linear Baseline: When ML Actually Helps

It's tempting to reach straight for gradient boosting or a neural network because "ML should find patterns humans and simple models miss." In finance, that instinct is often wrong, or at least premature: return data is extremely noisy relative to the signal in it, and a flexible model given that much noise will happily fit patterns that are pure chance rather than real structure. A simple linear regression, fit on the same features, is cheap, stable, and — crucially — hard to overfit by accident. If a complex model can't beat it out-of-sample, the complexity isn't paying for itself; it's just adding variance.

Why the baseline comes first

Think of the linear model as a control group in an experiment. Without it, you have no way to know whether a fancier model's apparent edge is real or just noise the more flexible model was better at memorizing. A random forest that scores 0.04 information coefficient sounds impressive in isolation — until you learn the linear regression, on the exact same features, scores 0.038. The complex model added enormous flexibility and computation for a difference well within noise, and it likely carries more overfitting risk and less interpretability for that marginal, possibly illusory gain.

Formally, out-of-sample error decomposes into bias and variance:

$$\mathbb{E}[(\hat{y} - y)^2] = \text{Bias}^2 + \text{Variance} + \text{noise}.$$

In plain English: a flexible model can reduce bias (it can represent more complicated relationships) but only if there's enough real signal and enough data to estimate that complexity without inflating variance (fitting noise instead). In financial cross-sections, the noise term is typically large and the true relationships are often close to linear or driven by a small number of known nonlinear effects — conditions where the variance cost of extra flexibility frequently exceeds the bias benefit.

Worked example: three models on the same features

A researcher builds a next-month stock-return model on ten standard factors (value, momentum, quality, size, and similar) and compares three models by out-of-sample information coefficient over a five-year walk-forward test: linear regression scores 0.031, a random forest scores 0.033, and a gradient-boosted tree scores 0.029. The random forest's edge over linear (0.033 vs. 0.031) is small and, checked across sub-periods, not consistent — it wins in some years and loses in others, which is the signature of noise rather than a real, exploitable nonlinearity. The gradient-boosted model actually underperforms the linear baseline, likely overfitting despite regularization. The honest conclusion is that these ten factors mostly enter returns close to linearly, and the added complexity of tree ensembles isn't earning its cost here — a different situation from, say, order-book microstructure features, where genuine threshold effects and interactions are common and nonlinear models often do add real value.

What this means in practice

The right workflow is always: fit the linear baseline first, on the same data and same walk-forward protocol as any candidate ML model, and require a candidate to beat it out-of-sample by a margin that survives sub-period checks and reasonable statistical scrutiny before adopting the extra complexity. ML tends to earn its keep when there's a real, known source of nonlinearity (regime-dependent effects, threshold behavior, rich interaction structure) and enough data to estimate it — not as a default upgrade applied to every problem regardless of what the underlying relationships look like.