Null Importance and Shadow Features

A tree-based model's built-in feature importance scores can be misleading: high-cardinality or noisy features sometimes get inflated importance just because trees find it easy to overfit to their many possible split points, not because they carry real signal. Null importance testing addresses this by establishing a baseline for what "importance by pure chance" looks like.

The procedure: shuffle the target variable randomly (destroying any real relationship between features and target), retrain the model, and record each feature's importance under this "null" scenario. Repeat this many times to build a distribution of null importances per feature. A feature is considered genuinely useful only if its importance on the real, unshuffled target sits well above the null distribution it produces when there's nothing real to learn.

Worked example

A high-cardinality "customer ID" feature scores as the top-importance feature in a gradient-boosted model. Running null importance testing, the target is shuffled 50 times and the model retrained each time; "customer ID" still scores highly even with a scrambled target, because trees can exploit its many unique values to overfit regardless of any real signal. Its real-target importance falls inside this null-shuffled range, revealing it's not actually predictive and should be dropped, while a feature whose real-target importance sits far above its null range is kept.