Bias in Random Forest Feature Importance

You train a random forest to predict which trades will be profitable, and the built-in "feature importance" ranking puts a near-meaningless feature — say, an internal trade ID or a continuous timestamp — near the top, above a genuinely predictive categorical flag like "venue". Nothing is wrong with your data pipeline. This is a known artifact of how the default importance score is computed, and it can quietly mislead every feature-selection decision built on top of it.

Where the bias comes from

A random forest's default impurity-based importance (often called "Gini importance") sums up, across every tree, how much each feature reduces impurity whenever it's chosen as a split. The problem is that a feature with many possible split points — a continuous variable, or a categorical variable with many distinct levels — gets many more candidate splits to choose from at every node. Even if that feature carries no real signal, the tree-growing algorithm searches over more cut points for it and, purely by chance, is more likely to find one that looks good on the training sample. A binary flag has exactly one possible split; a continuous feature with thousands of unique values has thousands.

In plain English: impurity importance measures how often and how effectively a feature was used to split, and features with more ways to be split get used more, whether or not they're actually informative.

Worked example: an uninformative ID column

Suppose you fit a forest on a dataset with 10 real features plus an accidental "row ID" column — a unique sequential number with no relationship to the target. Because the ID has as many distinct values as there are rows, at almost every node the tree can find some threshold on it that happens to separate a slightly higher proportion of positive labels in that bootstrap sample — pure noise the split-search algorithm is happy to exploit given effectively unlimited threshold choices. Across hundreds of trees, this adds up to a nontrivial importance score, sometimes rivaling real predictors, even though permuting the ID column would show its true importance is roughly zero.

A more reliable alternative

Permutation importance sidesteps this bias: instead of counting splits, it measures how much the model's actual predictive accuracy drops when a single feature's values are randomly shuffled, breaking its relationship with the target while leaving everything else intact. A feature with no real signal shows almost no accuracy drop when permuted, regardless of how many split candidates it offered during training. The tradeoff is cost: it requires re-scoring the model once per feature, whereas impurity importance is essentially free as a byproduct of training.

Drag the complexity slider in the explorer above and notice how a model with more capacity to carve up the feature space can fit noise in high-cardinality features the same way a tree can find a spurious split — more flexibility means more opportunity to overfit to nothing.

What this means in practice

Never trust a default "feature importance" plot from a random forest library without checking whether it's impurity-based or permutation-based — most libraries default to the cheaper, biased impurity version. If you're doing feature selection for a live trading signal, a biased importance ranking can lead you to keep a spurious, high-cardinality feature (like an exchange timestamp) and discard a genuinely predictive categorical one, degrading the model in ways that are hard to notice until it fails out-of-sample.