STRONG ORACLE OPTIMALITY OF FOLDED CONCAVE PENALIZED ESTIMATION

By Jianqing Fan, Lingzhou Xue, Hui Zou. Annals of Statistics, 2012

330 citations recorded by the source corpus.

Folded concave penalization methods have been shown to enjoy the strong oracle property for high-dimensional sparse estimation. However, a folded concave penalization problem usually has multiple local solutions and the oracle property is established only for one of the unknown local solutions. A challenging fundamental issue still remains that it is not clear whether the local optimum computed by a given optimization algorithm possesses those nice theoretical properties. To close this important theoretical gap in over a decade, we provide a unified theory to show explicitly how to obtain the oracle solution via the local linear approximation algorithm. For a folded concave penalized estimation problem, we show that as long as the problem is localizable and the oracle estimator is well behaved, we can obtain the oracle estimator by using the one-step local linear approximation. In addition, once the oracle estimator is obtained, the local linear approximation algorithm converges, namely it produces the same estimator in the next iteration. The general theory is demonstrated by using four classical sparse estimation problems, i.e., sparse linear regression, sparse logistic regression, sparse precision matrix estimation and sparse quantile regression.

Subfields: Computational Mechanics, Numerical Analysis, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/132daa9be756ce5ac7caa4d43847f63bc36d2124

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Numerical Analysis

In Statistics and Probability

Citations in the corpus

Uniform Inference for High-dimensional Quantile Regression: Linear Functionals and Regression Rank Scores (2017, background): “Approaches with non convex penalties (Fan and Li, 2001; Zhang, 2010; Fan and Lv, 2011; Fan et al., 2014; Wang et al., 2013) provide viable alternatives, however, these procedures rely on oracle properties and, as such, do not lead to uniform inference.”
Estimating structured high-dimensional covariance and precision matrices: Optimal rates and adaptive estimation (2016, methodology): “For estimation of sparse precision matrices, Professor Zou points out a twostep approach proposed in Fan et al. (2014). This hybrid method combines the two major approaches discussed in the current paper, namely neighborhoodbased CLIME and penalized likelihood approaches, and enjoys the merits of bo”
Machine Learning Advances for Time Series Forecasting (2020, methodology): “Unlike OLS or LASSO, we have to solve a nonconvex optimization problem that may have multiple minima and is computationally more intensive than the LASSO.”
Multi-Stage Multi-Task Feature Learning (2012, neutral): “However, a convex regularizer is known to be too loose to approximate the l0-type one and often achieves suboptimal performance (either require restrictive conditions or obtain a suboptimal error bound) (Zou and Li, 2008; Zhang, 2010, 2012; Zhang and Zhang, 2012; Shen et al., 2012; Fan et al., 2012)”
High‐dimensional quantile regression: Convolution smoothing and concave regularization (2021, background): “(2015), Wainwright (2019), Fan et al. (2020), and the references therein. In the context of QR, Belloni and Chernozhukov (2011) provided a comprehensive analysis of the l1-penalized QR as well as post-penalized QR estimator. Since then, the literature on high-dimensional QR has grown rapidly, and we”
Guaranteed Matrix Completion via Non-Convex Factorization (2014, neutral): “1.5.3].”
Text as Data (2017, methodology): “These diminishing-bias penalties lead to increased computation costs (due to a non-convex loss), but there exist efficient approximation algorithms (see, e.g., Fan, Xue, and Zou 2014; Taddy 2017b). sample standard deviation of that covariate.”
A Nonparametric Bayesian Analysis of Heterogenous Treatment Effects in Digital Experimentation (2014, methodology): “ can be used to get even closer to the L 0 ideal. For example, the adaptive lasso of Zou (2006) uses weights proportional to the inverse of a guess for the optimal solution, say w j = 1=j^ jj, while Fan et al. (2014) sets w j through 1-2 steps of an iterative solver targeting a concave penalized fun”
A General Theory of Hypothesis Tests and Confidence Regions for Sparse High Dimensional Models (2014, background): “In an early work, Knight and Fu (2000) showed that the limiting distribution of the Lasso estimator is not normal even in the low dimensional setting.”