Regularization Parameter Selections via Generalized Information Criterion

By Yiyun Zhang, Runze Li, Chih-Ling Tsai. Journal of the American Statistical Association, 2010

312 citations recorded by the source corpus.

We apply the nonconcave penalized likelihood approach to obtain variable selections as well as shrinkage estimators. This approach relies heavily on the choice of regularization parameter, which controls the model complexity. In this paper, we propose employing the generalized information criterion, encompassing the commonly used Akaike information criterion (AIC) and Bayesian information criterion (BIC), for selecting the regularization parameter. Our proposal makes a connection between the classical variable selection criteria and the regularization parameter selections for the nonconcave penalized likelihood approaches. We show that the BIC-type selector enables identification of the true model consistently, and the resulting estimator possesses the oracle property in the terminology of Fan and Li (2001). In contrast, however, the AIC-type selector tends to overfit with positive probability. We further show that the AIC-type selector is asymptotically loss efficient, while the BIC-type selector is not. Our simulation results confirm these theoretical findings, and an empirical example is presented. Some technical proofs are given in the online supplementary material.

Subfields: Artificial Intelligence, Control and Systems Engineering, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/77c69a423b61dfdf87a0de439a1232ea1c3a0192

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Statistics and Probability

Citations in the corpus

Tuning parameter selection in high dimensional penalized likelihood (2013, methodology): “Using the same notations as those in Zhang et al. (2010), we partition the interval [λmin, λmax] into subsets”
Model Selection via Bayesian Information Criterion for Quantile Regression Models (2014, background): “Zhang, Li, and Tsai (2010) showed that BIC can play the same role in penalized likelihood settings.”
Informative Estimation and Selection of Correlation Structure for Longitudinal Data (2012, methodology): “…information criterion (BIC) to select the tuning parameter of the SCAD penalty in penalized least squares for consistent model selection, and Zhang, Li, and Tsai (2010) proposed the generalized information criterion (GIC-type) for tuning parameter selection in a nonconcave penalized…”
Shrinkage estimation of common breaks in panel data models via adaptive group fused Lasso (2016, unknown): “Let ̃2T̃̃1 ≡ 1−11 (α̃̃1 ; T̃̃1 ) Following Wang et al. (2007), Zhang et al. (2010) and Su and Qian (2014), we propose to select the tuning parameter 1 by minimizing the following information criterion (IC): 1 (1) = ̃ 2 T̃̃1 + 1 (̃1 + 1) (3.3) where 1 is a tuning parameter which plays a similar…”
Tuning Parameter Selection for the Adaptive Lasso Using ERIC (2015, methodology): “This result was not surprising—in the fixed p setting at least, it has been shown that using crossvalidation to choose λ is asymptotically loss efficient rather than consistent (Wang, Li, and Tsai 2007; Zhang, Li, and Tsai 2010).”
Efficiency for Regularization Parameter Selection in Penalized Likelihood Estimation of Misspecified Models (2013, methodology): “In the context of generalized linear models (GLMs), Zhang et al. (2010) (hereafter ZLT) proposed the use of a “GIC-type” criterion, GICκn = − 1 n l(β̂λ) + κn dfλ n for choosing the regularization parameter λ for non-concave penalized estimators in both the non-true model world and the true-model wor”
Introduction to the LASSO (2018, background): “com [18] Yiyun Zhang Runze Li and Chih-Ling Tsai, Regularization Parameter selections via Generalized Information Criterion, Journal of the American Statistical Association, 105(489):312323, 2010.”
Variable selection using pseudo-variables (2018, background): “…there is an extensive literature on tuning parameter selection including information-based criteria (Chen and Chen, 2008; Wang et al., 2009; Zhang et al., 2010; Fan and Tang, 2013; Hui et al., 2015), resampling methods (Hall et al., 2009; Meinshausen and Bühlmann, 2010; Feng and Yu, 2013;…”