© Institute of Mathematical Statistics, 2004 LEAST ANGLE REGRESSION
By Bradley Efron, Trevor J. Hastie, Iain M. Johnstone, Robert Tibshirani
6,720 citations recorded by the source corpus.
The purpose of model selection algorithms such as All Subsets, Forward Selection and Backward Elimination is to choose a linear model on the basis of the same set of data to which the model will be applied. Typically we have available a large collection of possible covariates from which we hope to select a parsimonious set for the efficient prediction of a response variable. Least Angle Regression (LARS), a new model selection algorithm, is a useful and less greedy version of traditional forward selection methods. Three main properties are derived: (1) A simple modification of the LARS algorithm implements the Lasso, an attractive version of ordinary least squares that constrains the sum of the absolute regression coefficients; the LARS modification calculates all possible Lasso estimates for a given problem, using an order of magnitude less computer time than previous methods. (2) A different LARS modification efficiently implements Forward Stagewise linear regression, another promising new model selection method; this connection explains the similar numerical results previously observed for the Lasso and Stagewise, and helps us understand the properties of both methods, which are seen as constrained versions of the simpler LARS algorithm. (3) A simple approximation for the degrees of freedom of a LARS estimate is available, from which we derive a Cp estimate of prediction error; this allows a principled choice among the range of possible LARS estimates. LARS and its variants are computationally efficient: the paper describes a publicly available algorithm that requires only the same order of magnitude of computational effort as ordinary least squares applied to the full set of covariates.
Subfields: Control and Systems Engineering, Statistics and Probability, Optimization & Training, Statistical Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Control and Systems Engineering
- Robert Tibshirani
- Francis R. Bach
- Sergey Levine
- Nicolai Meinshausen
- Peter Buhlmann
- Jan Peters
- Cun-Hui Zhang
- Pieter Abbeel
In Statistics and Probability
- Robert Tibshirani
- Martin J. Wainwright
- Peter Buhlmann
- Jianqing Fan
- Francis R. Bach
- Cun-Hui Zhang
- Sara A. van de Geer
- Nicolai Meinshausen
In Optimization & Training
- Francis R. Bach
- Peter Richtárik
- Martin Jaggi
- Simon Lacoste-Julien
- Julien Mairal
- Michael I. Jordan
- Nicolai Meinshausen
- Jason D. Lee
In Statistical Learning Theory
- Martin J. Wainwright
- Francis R. Bach
- Cun-Hui Zhang
- Nicolai Meinshausen
- Peter Buhlmann
- Robert Tibshirani
- Tong Zhang
- Sara A. van de Geer
Citations in the corpus
- Minding the Gaps for Block Frank-Wolfe Optimization of Structured SVMs (2016, methodology): “According to the definition of Efron et al. (2004), a regularization path is a set of minimizers of a regularized objective in the form of (1) for all possible values of the regularization parameter λ. Similarly to LASSO and binary SVM, the general result of Rosset & Zhu (2007, Proposition 1) is…”
- Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition (2010, methodology): “ctive that penalizes the reconstruction error using a linear basis set and the sparsity of the corresponding representation. Many recent works have focused on efficiently solving the problem in eq. (2) [12, 5, 7, 13, 14, 6]. Yet, inference requires running some sort of iterative minimization algorithm”
- Supersparse linear integer models for optimized medical scoring systems (2015, methodology): “Further, many current methods also control accuracy and sparsity by means of convex surrogate functions to preserve scalability (Tibshirani, 1996, Efron et al., 2004, see e.g.).”
- Supersparse Linear Integer Models for Predictive Scoring Systems (2013, methodology): “[Efron et al. 2004] Efron, B.; Hastie, T.; Johnstone, I.; and Tibshirani, R. 2004.”
- Regularization Paths for Generalized Linear Models via Coordinate Descent (2010, methodology): “In some of the extensions above (items 2,3, and 6), piecewise-linearity can be exploited as in Efron et al. (2004) to yield efficient algorithms.”
- Distributed GraphLab: A Framework for Machine Learning in the Cloud (2012, background): “For example, in parameter optimization, often a large number of parameters will quickly converge in a few iterations, while the remaining parameters will converge slowly over many iterations [11, 10].”
- SIMULTANEOUS ANALYSIS OF LASSO AND DANTZIG SELECTOR (2008, methodology): “We refer to [9, 18, 19, 22] for detailed discussion of these optimization problems and fast algorithms.”
- Optimization with Sparsity-Inducing Penalties (2011, methodology): “Combined with the square loss, it leads to the group Lasso formulation [142, 156].”
- Component selection and smoothing in multivariate nonparametric regression (2006, neutral): “(2001); Efron, Hastie, Johnstone and Tibshirani (2004) ].”
- Methods and Models for Interpretable Linear Classification (2014, background): “The interpretability of transparent models is usually improved by tuning sparsity [107, 120, 29, 50, 89, 19, 59, 47], by ensuring monotonic relationships between certain input variables and the predicted outcome [10, 87, 114, 77], and by restricting coefficients to a small set of integer values [23,”
- Parallel Coordinate Descent for L1-Regularized Loss Minimization (2011, methodology): “We also tested published implementations of the classic algorithms GLMNET (Friedman et al., 2010) and LARS (Efron et al., 2004).”
- Nearly unbiased variable selection under minimax concave penalty (2010, methodology): “In each step, the PLUS computes one line segment in its path between two turning points, and its computational cost is the same as the LARS [Efron et al. (2004)] per step.”
- High-dimensional graphs and variable selection with the Lasso (2006, methodology): “The complexity of the proposed neighborhood selection for one node with the Lasso is reduced further to O(npmin{n,p}), as the Lars procedure of ( Efron et al., 2004 ) requires O(min{n,p}) steps, each of complexity O(np).”