Nearly unbiased variable selection under minimax concave penalty
By Cun-Hui Zhang. 2010
4,012 citations recorded by the source corpus.
We propose MC+, a fast, continuous, nearly unbiased and accurate method of penalized variable selection in high-dimensional linear regression. The LASSO is fast and continuous, but biased. The bias of the LASSO may prevent consistent variable selection. Subset selection is unbiased but computationally costly. The MC+ has two elements: a minimax concave penalty (MCP) and a penalized linear unbiased selection (PLUS) algorithm. The MCP provides the convexity of the penalized loss in sparse regions to the greatest extent given certain thresholds for variable selection and unbiasedness. The PLUS computes multiple exact local minimizers of a possibly nonconvex penalized loss function in a certain main branch of the graph of critical points of the penalized loss. Its output is a continuous piecewise linear path encompassing from the origin for infinite penalty to a least squares solution for zero penalty. We prove that at a universal penalty level, the MC+ has high probability of matching the signs of the unknowns, and thus correct selection, without assuming the strong irrepresentable condition required by the LASSO. This selection consistency applies to the case of p » n, and is proved to hold for exactly the MC+ solution among possibly many local minimizers. We prove that the MC+ attains certain minimax convergence rates in probability for the estimation of regression coefficients in l r balls. We use the SURE method to derive degrees of freedom and C p -type risk estimates for general penalized LSE, including the LASSO and MC+ estimators, and prove their unbiasedness. Based on the estimated degrees of freedom, we propose an estimator of the noise level for proper choice of the penalty level. For full rank designs and general sub-quadratic penalties, we provide necessary and sufficient conditions for the continuity of the penalized LSE. Simulation results overwhelmingly support our claim of superior variable selection properties and demonstrate the computational efficiency of the proposed method.
Subfields: Computational Mechanics, Statistics and Probability, Mathematical Optimization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Martin J. Wainwright
- Cun-Hui Zhang
- Peter Buhlmann
- Sara A. van de Geer
- Jianqing Fan
- Robert Tibshirani
- Tong Zhang
- Alexandre B. Tsybakov
In Statistics and Probability
- Peter Buhlmann
- Jianqing Fan
- Cun-Hui Zhang
- Martin J. Wainwright
- Sara A. van de Geer
- Victor Chernozhukov
- Bin Yu
- Nicolai Meinshausen
In Mathematical Optimization
- Martin J. Wainwright
- Peter Buhlmann
- Robert Tibshirani
- Bin Yu
- Trevor J. Hastie
- Jian Huang
- Jianqing Fan
- Hui Zou
Citations in the corpus
- Scheduling Policies for Federated Learning in Wireless Networks (2019, background): “Common choices for r ( w ) include the L-2 penalty k w k 22 , the L-1 penalty k w k 1 , or a family of folded concave functions [25].”
- Spectral Regularization Algorithms for Learning Large Incomplete Matrices (2010, background): “In penalized regression there have been recent developments directed towards “bridging” the gap between the l1 and l0 penalties (Friedman, 2008; Zhang, 2010; Mazumder et al., 2009).”
- On Nonconvex Decentralized Gradient Descent (2016, background): “Besides q quasi-norm, there are some other typical cases like SCAD [16] and MCP [68] widely used in statistical learning, satisfying the condition (c) of this theorem.”
- MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels (2017, background): “When θ = 1 it is equivalent to the minimax concave penalty (Zhang, 2010).”
- A General Theory of Concave Regularization for High-Dimensional Sparse Estimation Problems (2011, methodology): “For the MCP, the local minima generated by the path following the algorithm controls the estimation error and model size in the sense of (9) under an 2 regularity condition on X [42].”
- High-Dimensional Statistics with a View Toward Applications in Biology (2014, background): “Many versions of the Lasso have been proposed (Zou, 2006; Meinshausen, 2007; Zou and Li, 2008; van de Geer et al., 2011), and there are other penalized estimators which lead to sparse solutions (Fan and Li, 2001; Zhang, 2010).”
- High‐dimensional macroeconomic forecasting and variable selection via penalized regression (2019, methodology): “…model can be estimated by a penalized regression, such as the Lasso by Tibshirani (1996) and Frank and Friedman (1993) as a special case, the smoothly clipped absolute deviation (SCAD) penalized regression by Fan and Li (2001), and regression with the minimax concave penalty (MCP) by Zhang (2010).”
- Goodness‐of‐fit tests for high dimensional linear models (2015, methodology): “…is similar to the most challenging signal settings that were considered in Dezeure et al. (2015) and also resembles the estimated signal from regression of the true response associated with the gene expression data onto the predictors by using the lasso or the minimax concave penalty (Zhang, 2010).”
- Accelerated Proximal Gradient Methods for Nonconvex Programming (2015, methodology): “To address this issue, many nonconvex regularizers have been proposed, such as l p -norm [4], Capped-l 1 penalty [3], Log-Sum Penalty [2], Minimax Concave Penalty [5], Geman Penalty [6], Smoothly Clipped Absolute Deviation [7] and Schatten-p norm [8].”
- Minimization Over the Nonconvex Sparsity Constraint Using A Hybrid First-order method (2021, background): “To address this, researchers have explored nonconvex surrogates for the ℓ 0 norm, such as Cappled L 1 [51], Smoothly Clipped Absolute Deviation (SCAD)[18] and Minimax Concave Penalty (MCP)[48], as alternative solutions, aiming to develop more effective and efficient algorithms.”
- Should Regularization Replace Simple Structure Rotation in Exploratory Factor Analysis? (2019, background): “Extensive empirical comparisons of factor rotation techniques for typical psychometric data sets are available (Asparouhov & Muthén, 2009; Sass & Schmitt, 2010; Schmitt & Sass, 2011) and a number of studies have compared different penalty functions in the context of both exploratory factor analysis ”
- Accounting for linkage disequilibrium in genome-wide association studies: A penalized regression method (2013, background): “Zhang [21] proposed a flexible minmax concave penalty (MCP) which attenuates the effect of shrinkage that leads to bias.”
- Identification of Partially Linear Structure in Additive Models with an Application to Gene Expression Prediction from Sequences (2012, result): “Other choices of penalty, such as adaptive lasso (Zou, 2006) or minimax concave penalty (Zhang, 2010), are expected to produce similar results in both theory and practice.”
- Estimation of oblique structure via penalized likelihood factor analysis (2013, methodology): “For example, the MC+ (Zhang 2010) is given by For each value of ρ > 0, γ → ∞ yields soft threshold operator (i.e., lasso penalty) and γ → 1+ produces hard threshold operator.”
- Nonparametric Independence Screening in Sparse Ultra-High Dimensional Additive Models (2009, methodology): “The statistical literature contains numerous procedures on the variable selection for linear models and other parametric models, such as the Lasso (Tibshirani, 1996), the SCAD and other folded-concave penalty (Fan, 1997; Fan and Li, 2001), the Dantzig selector (Candes and Tao, 2007), the Elastic net”