One-step Sparse Estimates in Nonconcave Penalized Likelihood Models

By Hui Zou, Runze Li. 2008

1,287 citations recorded by the source corpus.

Fan & Li (2001) propose a family of variable selection methods via penalized likelihood using concave penalty functions. The nonconcave penalized likelihood estimators enjoy the oracle properties, but maximizing the penalized likelihood function is computationally challenging, because the objective function is nondifferentiable and nonconcave. In this article we propose a new unified algorithm based on the local linear approximation (LLA) for maximizing the penalized likelihood for a broad class of concave penalty functions. Convergence and other theoretical properties of the LLA algorithm are established. A distinguished feature of the LLA algorithm is that at each LLA step, the LLA estimator can naturally adopt a sparse representation. Thus we suggest using the one-step LLA estimator from the LLA algorithm as the final estimates. Statistically, we show that if the regularization parameter is appropriately chosen, the one-step LLA estimates enjoy the oracle properties with good initial estimators. Computationally, the one-step LLA estimation methods dramatically reduce the computational cost in maximizing the nonconcave penalized likelihood. We conduct some Monte Carlo simulation to assess the finite sample performance of the one-step sparse estimation methods. The results are very encouraging.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Best Subset Selection via a Modern Optimization Lens (2015, methodology): “To address the shortcomings, non-convex penalized regression is often used to “bridge” the gap between the convex `1 penalty and the combinatorial `0 penalty [39, 28, 25, 55, 56, 29, 62, 63, 58, 14].”
Nearly unbiased variable selection under minimax concave penalty (2010, methodology): “4) in Zou (2006) and Zou and Li (2008). Marginal regression x7y/||x7 ||2 can be used as an initial es timate of fij and is proved to result in the selection consistency of the adaptive LASSO under a certain partial orthogonality condition on the pairwise correla tions among vectors {y, xi,.”
Multi-Stage Multi-Task Feature Learning (2012, neutral): “However, a convex regularizer is known to be too loose to approximate the l0-type one and often achieves suboptimal performance (either require restrictive conditions or obtain a suboptimal error bound) (Zou and Li, 2008; Zhang, 2010, 2012; Zhang and Zhang, 2012; Shen et al., 2012; Fan et al., 2012)”
High-Dimensional Sparse Additive Hazards Regression (2012, background): “…Friedman, and Hastie (2011), generalized this idea to regularized regression with concave penalties and showed that it is an attractive alternative to earlier proposals such as the local quadratic approximation (LQA) (Fan and Li 2001) and local linear approximation (LLA) (Zou and Li 2008).”
Robust rank correlation based screening (2010, methodology): “As Ln(β) is a smoothing function of β, traditional optimal methods, such as the Newton Raphson algorithm or newly developed LARS [Efron et al. (2004)] and LLA [Zou and Li (2008)], can be used to obtain the maximizer of Ln(β) to simultaneously achieve the selection and estimation of β.”
Online Adaptive Estimation of Sparse Signals: Where RLS Meets the $\ell_1$ -Norm (2010, background): “Neglecting constant terms, the cost function in (6) can be re-written as (7) where (8) Due to data windowing, and can be updated recursively as [cf. (w1)–(w3)] (w1): (9a) (w2): (9b) (w3): (9c) Relative to the batch Lasso in (5), any of the TWL updates in (9) offers memory savings.”
Regression shrinkage and selection via the lasso: a retrospective (2011, neutral): “This has led to considerable recent research investigating other forms of penalty that allow for oracle properties (Zou, 2006; Zou and Li, 2008; Candès et al., 2008).”
On constrained and regularized high-dimensional regression (2013, methodology): “In a situation as such, little is known about selection consistency for many methods in terms of ( p , n ) -asymptotics as n , p → ∞ , although some methods such as adaptive Lasso (Zou 2006; Zou and Li 2008) have been examined for ﬁxed p -asymptotics as n → ∞ .”
A Selective Overview of Variable Selection in High Dimensional Feature Space (2009, methodology): “Fan and Li (2001), Zou (2006), and Zou and Li (2008) all suggest a consistent estimate such as the un-penalized MLE.”
Regularized M-estimators with nonconvexity: statistical and algorithmic theory for local optima (2013, methodology): “Various methods have been proposed for optimizing convex loss functions with nonconvex penalties [3, 4, 15], but these methods are only guaranteed to generate local optima of the composite objective, which have not been proven to be well-behaved.”
Shrinkage tuning parameter selection with a diverging number of parameters (2009, methodology): “To save computational time, the one-step sparse estimator of Zou and Li (2008) is used for SCAD.”
Semiparametric regression models with additive nonparametric components and high dimensional parametric components (2012, methodology): “We use the following one-step approximation procedure modified from Zou and Li (2008).”
Asymptotic Equivalence of Regularization Methods in Thresholded Parameter Space (2013, methodology): “Algorithms for implementing regularization methods include those mentioned in the Intro-duction, the LQA algorithm (Fan and Li, 2001), and LLA algorithm (Zou and Li, 2008).”
Varying Coefficient Regression Models: A Review and New Developments (2015, methodology): “Inspired by Zou & Li (2008) and Noh & Park (2010), one may replace p .kˇj k/ in (2.6) by its onestep approximation p0 .k Ǒ .0/j k/kˇj k, where Ǒ .0/ j are suitable initial estimators of ˇj .”