Forward stagewise regression and the monotone lasso

By Trevor J. Hastie, Jonathan E. Taylor, Robert Tibshirani, Guenther Walther. 2007

133 citations recorded by the source corpus.

We consider the least angle regression and forward stagewise algorithms for solving penalized least squares regression problems. In Efron, Hastie, Johnstone & Tibshirani (2004) it is proved that the least angle regression algorithm, with a small modification, solves the lasso regression problem. Here we give an analogous result for incremental forward stagewise regression, showing that it solves a version of the lasso problem that enforces monotonicity. One consequence of this is as follows: while lasso makes optimal progress in terms of reducing the residual sum-of-squares per unit increase in $L_1$-norm of the coefficient $\beta$, forward stage-wise is optimal per unit $L_1$ arc-length traveled along the coefficient path. We also study a condition under which the coefficient paths of the lasso are monotone, and hence the different algorithms coincide. Finally, we compare the lasso and forward stagewise procedures in a simulation study involving a large number of correlated predictors.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/c3da3f5e369ec6219018bf45a1fef4a80e814e20

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Efficient Bayesian inference of subsurface flow models using nested sampling and sparse polynomial chaos surrogates (2014, background): “The details of the mathematical implementation of the algorithm can be found in [51,52].”
An Interior-Point Method for Large-Scale $\ell_1$-Regularized Least Squares (2007, background): “The Lasso problem (8) has the form (5) with the variables .”
1 Trend Filtering (2009, methodology): “In statistics, the idea of ℓ1 regularization is used in the well-known Lasso algorithm [71] for ℓ1-regularized linear regression, its extensions such as the fused Lasso [72], the elastic net [82], the grouped Lasso [80], and the monotone Lasso [38], and ℓ1-regularized logistic regression [46, 47, 59”
An Interior-Point Method for Large-Scale l1-Regularized Logistic Regression (2007, methodology): “…it is used in the well-known Lasso algorithm (Tibshirani, 1996) for 1 -regularized linear regression, and its extensions such as the fused Lasso (Tibshirani et al., 2005), the grouped Lasso (Kim et al., 2006; Yuan and Lin, 2006; Zhao et al., 2007), and the monotone Lasso (Hastie et al., 2007).”
Least angle and ℓ1 penalized regression: A review (2008, background): “Stagewise is closely related to boosting, or “slow learning” in machine learning (Efron et al., 2004; Hastie et al., 2007).”