Feature Crosses for Tabular Models

A linear model predicting next-day return from "momentum" and "sector" learns one coefficient for momentum and one for each sector, added together. That structure cannot express "momentum works well in Technology but backfires in Utilities" — a linear model with those two features separately can only shift the whole prediction up or down for a sector, not change how strongly momentum matters within it. The effect of momentum, in the model's eyes, is the same number no matter what sector you are in.

A feature cross fixes this by building a new feature out of the combination of two existing ones, so the model can assign a completely separate coefficient to each combination rather than one coefficient per feature.

How it works

For two categorical features, the simplest cross is Cartesian: a new category for every pairing. "Momentum decile" (1–10) crossed with "sector" (say, 5 sectors) produces 50 new categories, one per (decile, sector) pair, each one-hot encoded as its own column. A linear model fit on the crossed feature can now give the (high momentum, Technology) combination its own weight, entirely separate from (high momentum, Utilities).

For numeric features, a cross is usually just the product: momentum × volatility as a single new column. This lets a linear model express that momentum's effect scales with volatility, which the two features alone cannot do.

Worked example

Suppose the true relationship in the data is: momentum predicts positive next-day return in Technology (coefficient +0.6) but predicts negative return in Utilities (coefficient −0.4) — momentum in a low-volatility, rate-sensitive sector tends to mean-revert. A linear model with momentum and sector as separate features can only fit one coefficient for momentum, say an average like $+0.1$, missing both effects. Cross them into momentum_x_sector and the model fits two separate coefficients: $+0.6$ on the (momentum, Technology) column and $-0.4$ on the (momentum, Utilities) column, recovering the true relationship exactly, because each combination now gets its own free parameter.

Why tree ensembles need this less

Tree-based models split on one feature at a time but can chain splits — split on sector, then within the Technology branch split on momentum — which approximates an interaction automatically. This is why feature crosses matter enormously for linear models and much less for gradient-boosted trees or random forests, which is part of why tree ensembles are the default for tabular finance problems (see Gradient Boosting in Finance): they save the analyst from having to guess which interactions to hand-construct.

The cost: combinatorial sparsity

Crossing two features multiplies their cardinalities. A cross of two 20-category features already has 400 possible combinations; cross three such features and you can exceed the size of the dataset itself. Most combinations then have few or no observations, and any coefficient the model fits for a rarely-seen combination is mostly noise — the same Overfitting risk that high-cardinality one-hot encoding runs into, for the same reason: too many free parameters, too little data per parameter.

Crossing numeric features and choosing the granularity

For two continuous features, crossing does not require binning into deciles first — the plain product momentum × volatility is itself a valid cross, and it is often the more natural one when the interaction is genuinely multiplicative (momentum's effect literally scaling with volatility, rather than behaving differently within discrete volatility bands). Binning first and crossing the bins, as in the sector example above, trades away that smoothness for interpretability: a coefficient on the (high-momentum, high-volatility) bucket is something you can read and reason about directly, in a way a single coefficient on a continuous product term is not. The choice of how many bins to use is itself a bias-variance trade-off familiar from elsewhere in modelling — few, wide bins waste resolution and blur genuinely different behaviour together; many, narrow bins recreate the sparsity problem this whole topic exists to manage, with each bin holding too few observations to fit reliably.

Depth-two crosses and diminishing returns

Nothing stops you from crossing a cross — (momentum × sector) × volatility_regime — but the sparsity cost compounds multiplicatively at each additional layer, while the marginal signal gained from a third-order interaction is, in most financial datasets, much smaller than the signal in the first-order features and simple pairwise crosses. In practice, two-way crosses built from a small number of features chosen for a specific reason capture the bulk of the available benefit; three-way and higher crosses are rarely worth their cost outside of very large datasets, and tree ensembles reach that same higher-order structure automatically through chained splits without needing it engineered by hand at all.