Regularization Paths for Generalized Linear Models via Coordinate Descent
By Jerome H. Friedman, Trevor J. Hastie, Robert Tibshirani. 2010
18,574 citations recorded by the source corpus.
We develop fast algorithms for estimation of generalized linear models with convex penalties. The models include linear regression, two-class logistic regression, and multinomial regression problems while the penalties include ℓ1 (the lasso), ℓ2 (ridge regression) and mixtures of the two (the elastic net). The algorithms use cyclical coordinate descent, computed along a regularization path. The methods can handle large problems and can also deal efficiently with sparse features. In comparative timings we find that the new algorithms are considerably faster than competing methods.
Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Statistics and Probability, Mathematical Optimization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Francis R. Bach
- Peter Richtárik
- Martin Jaggi
- Robert Tibshirani
- Martin J. Wainwright
- Martin Takác
- Wotao Yin
- Michael I. Jordan
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Andrew Zisserman
- Jitendra Malik
- Andrea Vedaldi
In Statistics and Probability
- Stefan Wager
- Robert Tibshirani
- Peter Buhlmann
- Susan Athey
- Victor Chernozhukov
- Martin J. Wainwright
- Bin Yu
- Cun-Hui Zhang
In Mathematical Optimization
- Cynthia Rudin
- Robert Tibshirani
- Berk Ustun
- Stefan Wager
- Martin Jaggi
- Sara A. van de Geer
- Peter Buhlmann
- Bin Yu
Citations in the corpus
- Dropout Training as Adaptive Regularization (2013, methodology): “6 Dropout training has been empirically found to perform well on tasks such as document (5)Normalizing the columns of the design matrix before performing penalized regression is standard practice, and is implemented by default in software like glmnet for R [16].”
- CoCoA: A General Framework for Communication-Efficient Distributed Optimization (2016, methodology): “For the L 1 examples discussed, existing fast L 1 -solvers for the single-machine case, such as glmnet variants (Friedman et al., 2010) or blitz (Johnson and Guestrin, 2015) can be directly applied to each local subproblem ) within Algorithm 1.”
- Is Feature Selection Secure against Training Data Poisoning? (2015, methodology): “5 for the elastic net, and then optimized the regularization parameter λ for all methods by retaining the best value over the entire regularization path (Friedman et al., 2010; Pe-dregosa et al., 2011).”
- Explanations of model predictions with live and breakDown packages (2018, methodology): “This option allows for performance of variable selection based on LASSO implemented in glmnet ((Friedman et al., 2010), (Simon et al., 2011)).”
- Interpretable classification models for recidivism prediction (2015, methodology): “(0:0001;0:001;0:01) C5.0 Decision Trees C5.0T c50 (Kuhn et al., 2012) 19 values of W+ C5.0 Decision Rules C5.0R c50 (Kuhn et al., 2012) 19 values of W+ Logistic Regression (L 1-Penalty) Lasso glmnet (Friedman et al., 2010) 19 values of W+ 100 values of L 1-penalty chosen by glmnet Logistic Regressio”
- Predictive Multiplicity in Classification (2019, methodology): “e multiplicity can still reveal salient information – even without using specialized techniques. We construct a pool of linear classifiers via penalized logistic regression using theglmnet package of [17]. Each model is trained using logistic regression along with a weighted combination of the ℓ 1 a”
- Generalized Sparse Additive Models (2019, background): “In many cases, this algorithm has a per-iteration complexity of O ( np ) — precisely that of state-of-the-art algorithms for the lasso [Friedman et al., 2010, Beck and Teboulle, 2009b].”
- Learning Optimized Risk Scores (2016, methodology): “We train PLR models using the glmnet package of Friedman et al. (2010). The coefficients of each model are the solution to the optimization problem:”
- Supersparse linear integer models for optimized medical scoring systems (2015, methodology): “+ 2 penalty Ridge glmnet (Friedman et al. 2010) 1000 values of λ chosen by glmnet”
- Sparse Partially Linear Additive Models (2014, methodology): “j;1=t0 j gand = minfC j;1=t0 j g. Thus, according to Theorem 2 in [37], Algorithm 1 converges Q-linearly. A.4 Practical Issues A.4.1 Active Set Strategy We employ the widely used active set strategy [39, 40, 41]. After a complete cycle through all the variables, we iterate only on the active set til”
- Local Linear Forests (2018, methodology): “Lasso is implemented via glmnet [Friedman et al., 2010] and is cross-validated with their automatic cross-validation feature.”
- Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso (2017, positive): “In our simulations, we use coordinate descent with warm starts over a sequence of tuning parameter values λ1 > · · · > λm > 0, as implemented in the glmnet R package (Friedman et al., 2007, 2010).”
- Approximate residual balancing: debiased inference of average treatment effects in high dimensions (2016, methodology): “This procedure is implemented in our R package balanceHD; we default to ζ=0:5 and α=0:9, and select λ by cross-validation using the lambda.1se rule from the glmnet package (Friedman et al., 2010).”
- Estimating treatment effect heterogeneity in randomized program evaluation (2013, methodology): “When the number of treatments K is large as in the GOTV study, a simple strategy of subsetting the data and conducting a separate analysis for each treatment suffers from the lack of power and multiple testing problems.”
- High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes (2019, methodology): “The parameters of equation 1 are estimated by the Lasso implemented in [4] using the procedure described in [8].”
- Coordinate descent algorithms (2015, methodology): “The same authors [19] apply CD to generalized linear models such as linear least squares and logistic regression, with convex regularization terms.”