A unified approach to model selection and sparse recovery using regularized least squares

By Jinchi Lv, Yingying Fan. 2009

376 citations recorded by the source corpus.

Model selection and sparse recovery are two important problems for which many regularization methods have been proposed. We study the properties of regularization methods in both problems under the unified framework of regularized least squares with concave penalties. For model selection, we establish conditions under which a regularized least squares estimator enjoys a nonasymptotic property, called the weak oracle property, where the dimensionality can grow exponentially with sample size. For sparse recovery, we present a sufficient condition that ensures the recoverability of the sparsest solution. In particular, we approach both problems by considering a family of penalties that give a smooth homotopy between L 0 and L 1 penalties. We also propose the sequentially and iteratively reweighted squares (SIRS) algorithm for sparse recovery. Numerical studies support our theoretical results and demonstrate the advantage of our new methods for model selection and sparse recovery.

Subfields: Computational Mechanics, Mathematical Physics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Jinchi Lv
2. Jianqing Fan
3. Hui Zou
4. Yingying Fan
5. Cheng Yong Tang
6. Wei Lin
7. Heng Peng
8. Bin Yu

In Mathematical Physics

1. Jinchi Lv
2. Yingying Fan
3. Jianqing Fan
4. Heng Peng
5. Cheng Yong Tang
6. Hui Zou
7. Bin Yu
8. Chenlei Leng

In Statistics and Probability

1. Jinchi Lv
2. Yingying Fan
3. Jianqing Fan
4. Cun-Hui Zhang
5. Chenlei Leng
6. Peter Buhlmann
7. Hui Zou
8. Bin Yu

Citations in the corpus

• A Framework for Feature Selection in Clustering (2010, background): “Other penalties on w could be used that would also yield sparsity (see, e.g., Fan and Li 2001; Lv and Fan 2009).”
• Parametric or nonparametric? A parametricness index for model selection (2011, background): “, [17, 27, 40, 44, 48, 50, 51, 52, 53, 56, 59, 63, 65, 69, 73, 76, 77]) have been established mostly in terms of either selection consistency or an asymptotic optimality.”
• High-Dimensional Sparse Additive Hazards Regression (2012, methodology): “…Journal of the American Statistical Association March 2013, Vol. 108, No. 501, Theory and Methods DOI: 10.1080/01621459.2012.746068 247 Ma (2007) proposed a weighted L1 approach and Martinussen and Scheike (2009) considered several regularization methods including the Lasso and Dantzig selector.”
• A Simple Two-Sample Test in High Dimensions Based on L2-Norm (2020, background): “In regression, feature screening and selection, and dimension reduction are the mainstream approaches (Cheng and Wu 2013; Cook and Li 2002; Fan and Li 2001; Fan and Lv 2008; Fan and Peng 2004; Fan and Song 2010; Lv and Fan 2009; Li et al. 2012; Tibshirani 1996; Xia 2007, 2008).”
• Penalized empirical likelihood and growing dimensional general estimating equations (2012, methodology): “…penalized likelihood appro ch for building such models; for example lasso (Tibshirani, 1996), the smoothly clipped absolute deviation method (Fan & Li, 2001), adaptive lasso (Zou, 2006; Zhang & Lu, 2007), least squares approximation (Wang & Leng, 2007), the folded concave penalty (Lv & Fan, 2009).”
• Robust rank correlation based screening (2010, neutral): “However, directly applying LASSO or Dantzig selector to ultra-high dimensional modeling is not a good choice because the irrepresentable conditions can be rather stringent in high dimensions; see, for example, Lv and Fan (2009) and Fan and Lv (2010).”
• A Selective Overview of Variable Selection in High Dimensional Feature Space (2009, background): “Lv and Fan (2009) establish a nonasymptotic weak oracle property for the PLS estimator.”
• Asymptotic Properties of Lasso+mLS and Lasso+Ridge in Sparse High-dimensional Linear Regression (2013, positive): “Replacing l1 penalty by l2 penalty in (2) gives the Ridge estimator [28]: β̂Ridge(λn) = argmin β { ||Y −Xβ||(2)2 + λn||β||(2)2 }”
• High dimensional thresholded regression and shrinkage effect (2014, methodology): “Other nonconvex regularization methods include the bridge regression using the L q -penalty for 0 q 1 (Frank and Friedman, 1993), the minimax concave penalty (MCP) (Zhang, 2010), and the smooth integration of counting and absolute deviation (SICA) penalty (Lv and Fan, 2009).”
• Nonparametric independence screening and structure identification for ultra-high dimensional longitudinal data (2013, methodology): “In the literature, the screening step is immediately followed by the model fitting step, thus usually some iterative screening procedure is employed to control the false selection rate [7, 21, 10, 3].”
• Decoupling Shrinkage and Selection in Bayesian Linear Models: A Posterior Summary Perspective (2014, background): “Specifically, the local linear approximation argument of Zou and Li (2008) and Lv and Fan (2009) advises to solve a surrogate optimization problem (for any wj near the corresponding 0 solution) βλ ≡ arg minγ ∑ j λ |wj | |γj | + n −1||Xβ̄ − Xγ ||22.”
• On constrained and regularized high-dimensional regression (2013, background): “Using the fact that I ≤ 1, we obtain that I ≤ ( 2 1−e−1 + 1 ) exp ( − n 18σ2 ( Cmin − 36 log p n σ 2 )) , leading to (10).”
• Non-Concave Penalized Likelihood with NP-Dimensionality (2009, methodology): “We set (n, p) = (200, 25) and chose the true regression coefficients vector β0 by setting β1 = (1.”
• The fused Kolmogorov filter: A nonparametric model-free screening method (2014, methodology): “It requires sophisticated estimation techniques, strong model assumptions and often advanced computing algorithms [Fan and Li (2001), Lv and Fan (2009), Tibshirani (1996), Zhang (2010)].”