Discovering Feature Interactions With Trees

A feature interaction is when two variables only matter to a prediction jointly — neither one, alone, tells you much, but together they carry real information. For example, a stock's momentum signal might only predict future returns well when trading volume is also high; on low-volume days the same momentum reading might be nearly meaningless. A linear regression, which adds up each feature's contribution independently, structurally cannot capture this unless someone explicitly creates a "momentum times volume" interaction term by hand — and with dozens of candidate features, manually trying every pairwise combination is impractical.

Decision trees find interactions automatically, almost for free, because of how they're built: each split conditions on the data that reached it through all prior splits. If a tree splits first on volume and then, within the high-volume branch, finds momentum highly useful for a further split — but finds momentum nearly useless within the low-volume branch — that pattern of splits is itself direct evidence of an interaction, discovered without anyone specifying it in advance.

In practice, quants exploit this by fitting a gradient-boosted tree ensemble on candidate features, then inspecting which feature pairs tend to co-occur along the same root-to-leaf paths, or using interaction-strength statistics computed from the fitted trees. A pair that shows up together disproportionately often is a strong hint to build an explicit interaction term for a simpler, more interpretable production model, rather than relying on the black-box tree ensemble alone.