Stepwise and Best Subset Selection

With $p$ candidate predictors, best subset selection fits a separate regression for every possible subset of them — $2^p$ models in total — and picks the best by a criterion like adjusted R-squared or AIC that penalizes extra variables. This is exhaustive and guaranteed to find the best-scoring subset of any given size, but becomes computationally impossible once $p$ climbs past the low twenties, since $2^{20}$ is already over a million models.

Stepwise selection is the practical workaround: instead of checking every subset, it builds the model greedily. Forward stepwise starts with no predictors and adds, at each step, whichever single remaining variable improves the fit the most, stopping once no addition helps enough. Backward stepwise starts with all predictors and removes the least useful one at each step. Both require only on the order of $p^2$ model fits rather than $2^p$, making them feasible for hundreds of candidate factors, at the cost of no longer being guaranteed to find the globally best subset — a greedy step taken early can lock out a combination that would have scored better overall.

In quant factor research, a researcher with 80 candidate signals might use forward stepwise to narrow down to a handful that jointly explain returns, understanding that the resulting model is a good, not provably optimal, subset, and cross-checking the survivors out of sample since stepwise procedures are especially prone to overfitting when many candidates are tried against the same data.