Discovering Feature Interactions With Trees

A feature interaction is when two variables only matter to a prediction jointly, neither one, alone, tells you much, but together they carry real information. For example, a stock's momentum signal might only predict future returns well when trading volume is also high; on low-volume days the same momentum reading might be nearly meaningless. A linear regression, which adds up each feature's contribution independently, structurally cannot capture this unless someone explicitly creates a "momentum times volume" interaction term by hand, and with dozens of candidate features, manually trying every pairwise combination is impractical.

Decision trees find interactions automatically, almost for free, because of how they're built: each split conditions on the data that reached it through all prior splits. If a tree splits first on volume and then, within the high-volume branch, finds momentum highly useful for a further split, but finds momentum nearly useless within the low-volume branch, that pattern of splits is itself direct evidence of an interaction, discovered without anyone specifying it in advance.

In practice, quants exploit this by fitting a gradient-boosted tree ensemble on candidate features, then inspecting which feature pairs tend to co-occur along the same root-to-leaf paths, or using interaction-strength statistics computed from the fitted trees. A pair that shows up together disproportionately often is a strong hint to build an explicit interaction term for a simpler, more interpretable production model, rather than relying on the black-box tree ensemble alone.