COORDINATE DESCENT ALGORITHMS FOR NONCONVEX PENALIZED REGRESSION, WITH APPLICATIONS TO BIOLOGICAL FEATURE SELECTION

By Patrick Breheny, Jian Huang. Annals of Applied Statistics, 2011

823 citations recorded by the source corpus.

A number of variable selection methods have been proposed involving nonconvex penalty functions. These methods, which include the smoothly clipped absolute deviation (SCAD) penalty and the minimax concave penalty (MCP), have been demonstrated to have attractive theoretical properties, but model fitting is not a straightforward task, and the resulting solutions may be unstable. Here, we demonstrate the potential of coordinate descent algorithms for fitting these models, establishing theoretical convergence properties and demonstrating that they are significantly faster than competing approaches. In addition, we demonstrate the utility of convexity diagnostics to determine regions of the parameter space in which the objective function is locally convex, even though the penalty is not. Our simulation study and data examples indicate that nonconvex penalties like MCP and SCAD are worthwhile alternatives to the lasso in many applications. In particular, our numerical results suggest that MCP is the preferred approach among the three methods.

Subfields: Artificial Intelligence, Computational Mechanics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/c67136ab0e52283d5b3beef23fe318eb9f455cb7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Statistics and Probability

Citations in the corpus

A Critical Review of LASSO and Its Derivatives for Variable Selection Under Dependence Among Covariates (2020, background): “− SCAD: ncvreg of Breheny and Huang (2011), last update July 9, 2020. − Adaptive LASSO (AdapL): glmnet of Friedman et al. (2010), last update June 16, 2020.”
A Split-and-Conquer Approach for Analysis of Extraordinarily Large Data (2014, methodology): “In order to get the SCAD and MCP estimators, the NCVREG algorithm (Breheny and Huang (2011)) was applied and a 10-fold cross-validation was used to select the tuning parameters.”
Review and evaluation of penalised regression methods for risk prediction in low‐dimensional data with few events (2015, methodology): “Breheny and Huang [27] compared SCAD with lasso in a gene expression and a genetic association study and noted that SCAD allows coefficients to take on large values much more easily than © 2015 The Authors.”
A variational Bayes approach to variable selection (2017, methodology): “We compare the performance of our VB method against the Lasso, SCAD and MCP penalized regression methods as implemented by the R package ncvreg [8].”
Tuning parameter selection in high dimensional penalized likelihood (2013, methodology): “Applying the AIC criterion, 7 genes were selected, which is close to the results by the cross-validation procedure applied in Breheny and Huang (2011).”
High‐dimensional macroeconomic forecasting and variable selection via penalized regression (2019, methodology): “All the computations are conducted using R 3.4.4 and we use ncvreg package of Breheny and Huang (2011) for the penalized regression and midasr for constructing MIDAS regressors.”
Regularized M-estimators with nonconvexity: statistical and algorithmic theory for local optima (2013, methodology): “In addition to nonconvex loss functions, our theory also applies to nonconvex regularizers, shedding new light on a long line of recent work involving the nonconvex SCAD and MCP regularizers [3, 2, 13, 14].”
Cox-nnet: An artificial neural network method for prognosis prediction of high-throughput omics data (2018, methodology): “Given the large amount of input features in gene expression data, penalization meth-ods such as LASSO (L1 norm), ridge (L2 norm) and MCP [2] regularizations are often used to help select representative feature in Cox-PH models.”
The Spike-and-Slab LASSO (2018, methodology): “(2011), ncvreg of Breheny and Huang (2011)), the Spike-and-Slab LASSO permits the use of a self-adaptive NSSL penalty, avoiding the need for tuning its complexity parameter θ.”
A Model-Averaging Approach for High-Dimensional Regression (2014, methodology): “MAIC, model averaging with the Akaike information criterion under the restriction ∑M k=1 wk = 1; MCV1, cross-validation model averaging without the restriction ∑M k=1 wk = 1 and the number of models and the number of regressors in each model is fixed; MCV2, cross-validation model averaging without t”
Estimation of oblique structure via penalized likelihood factor analysis (2013, background): “In particular, the coordinate descent algorithm is known as a remarkably fast algorithm (Friedman et al. 2010) and can also be applied to a wide variety of convex and nonconvex penalties (Breheny and Huang 2011; Mazumder et al. 2011).”
Group descent algorithms for nonconvex penalized linear and logistic regression models with grouped predictors (2012, methodology): “This algorithm was further improved upon by Breheny and Huang (2011), who demonstrated how a coordinate descent approach may used to ﬁt SCAD and MCP models in a very eﬃcient manner capable of scaling up to deal with very large problems.”
Coordinate descent algorithms (2015, background): “Breheny and Huang [10] discuss coordinate descent for linear and logistic regression with nonconvex separable regularization terms, reporting results for genetic association and gene expression studies.”
Multi-level Lasso for Sparse Multi-task Regression (2012, methodology): “We plan to characterize these regions as future work (in the spirit of (Breheny & Huang, 2011)).”
Efficiency for Regularization Parameter Selection in Penalized Likelihood Estimation of Misspecified Models (2013, methodology): “The Lasso regressions are fit using the R lars package (Hastie and Efron 2011) and the SCAD regressions are fit using the R ncvreg package (Breheny and Huang 2011). The lars package computes the entire regularization path for the Lasso and for SCAD the models are fit over a grid of 200 λ values from”
Model Selection via Bayesian Information Criterion for Quantile Regression Models (2014, methodology): “Recently, many efficient methods for finding the path have been developed such as the LARS algorithm (Efron et al. 2004) and the coordinate-descent based algorithms (Breheny and Huang 2011; Mazumder, Friedman, and Hastie 2011).”
Semiparametric Regression Pursuit (2012, methodology): “(1986), and Heckman (1986)), the partial residual estimator (Robinson (1988), Speckman (1985)), and polynomial spline estimator (Chen (1988)). An excellent discussion of partially linear models can be found in the book of Härdle Liang, and Gao (2000), which also contains an extensive list of referen”