Boruta and All-Relevant Feature Selection
Boruta is a feature selection method that keeps every feature genuinely more useful than random noise, rather than stopping at the smallest set that gets the job done.
Most feature selection methods look for the smallest, most predictive subset of features — a "minimal-optimal" set. Boruta does something different: it tries to find every feature that carries real signal, even redundant ones, which is called all-relevant feature selection.
It works by adding a "shadow" copy of every real feature to the dataset, with the shadow's values randomly shuffled so any relationship to the target is destroyed by construction. It then trains a random forest and compares each real feature's importance score to the best score among all the shadow (noise) features. A real feature that reliably beats the best shadow feature across many runs is kept as genuinely relevant; one that doesn't beat noise is dropped.
Boruta keeps a feature only if it consistently outperforms a deliberately randomized, noise-only copy of itself — this catches every useful feature, including redundant ones, rather than trimming down to a minimal predictive set.
Worked example
A dataset has 40 features, including one that's pure random noise. Boruta creates a shuffled shadow copy of all 40, trains a random forest on the combined 80 columns, and finds the shadow features' importance scores form a rough baseline "noise band." Real features scoring well above that band across repeated runs are confirmed as relevant; the genuine noise column scores within the shadow band and gets correctly rejected, along with any other feature that never clearly beats it.
Related concepts
Practice in interviews
Further reading
- Kursa & Rudnicki, 'Feature Selection with the Boruta Package', Journal of Statistical Software