Least Angle Regression

Least angle regression (LARS) fits a linear model by adding predictors gradually instead of solving one big optimization in one shot. It starts with no variables and an all-zero prediction, finds the single predictor most correlated with the target, and moves the coefficient in that variable's direction — not all the way to its full least-squares value, but only until some other predictor becomes just as correlated with the shrinking residual. At that point it adds the second variable and moves both together in the direction that keeps them equally correlated with the residual, repeating until every variable is in the model (or a stopping rule cuts it off). Each step is a specific, computable distance, so the whole path of coefficients from zero to the full least-squares fit comes out of one efficient pass, with roughly the same computational cost as a single ordinary least squares fit.

The name refers to the geometry: the direction of travel at each step makes an equal angle with every currently active predictor, which is exactly the condition for "equally correlated." A small modification to the algorithm — dropping a variable from the active set the moment its coefficient would cross zero — produces the exact lasso solution path, making LARS the standard way lasso is actually computed rather than solved by generic optimization.

For example, with predictors A and B and A more correlated with the target, LARS moves the A coefficient up until B's correlation with the residual catches up to A's, then moves both A and B together, tracing out a piecewise-linear path of coefficients as a function of the regularization strength.