A Selective Overview of Variable Selection in High Dimensional Feature Space

By Jianqing Fan, Jinchi Lv. 2009

967 citations recorded by the source corpus.

High dimensional statistical problems arise from diverse fields of scientific research and technological development. Variable selection plays a pivotal role in contemporary statistical learning and scientific discoveries. The traditional idea of best subset selection methods, which can be regarded as a specific form of penalized likelihood, is computationally too expensive for many modern statistical applications. Other forms of penalized likelihood methods have been successfully developed over the last decade to cope with high dimensionality. They have been widely applied for simultaneously selecting important variables and estimating their effects in high dimensional statistical inference. In this article, we present a brief account of the recent developments of theory, methods, and implementations for high dimensional variable selection. What limits of the dimensionality such methods can handle, what the role of penalty functions is, and what the statistical properties are rapidly drive the advances of the field. The properties of non-concave penalized likelihood and its roles in high dimensional statistical modeling are emphasized. We also review some recent advances in ultra-high dimensional variable selection, with emphasis on independence screening and two-scale methods.

Subfields: Artificial Intelligence, Computational Mechanics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Statistics and Probability

Citations in the corpus

Forward stability and model path selection (2021, neutral): “…on assessing consistency and establishing conditions under which it can be guaranteed that the estimated tuning parameter(s) cor-respondingtothechosen ˆ f m convergestotheirtrueoroptimal values [see e.g. Zhao and Yu (2006); Bach (2008); Zhang (2009) or Fan and Lv (2010) for a thorough overview].”
On asymptotically optimal confidence regions and tests for high-dimensional models (2013, background): “Other references and exposition to a broad class of models can be found in [18] or [10].”
Tuning parameter selection in high dimensional penalized likelihood (2013, methodology): “The Lasso method performs relative poorly, due to its bias issue (Fan and Lv, 2010).”
Model Selection in Linear Mixed Models (2013, methodology): “For the independent cluster model, mAIC is asymptotically equivalent to leave-one-cluster-out cross-validation using a marginal generalized least squares criterion (Fang (2011)); see Section 3.4. The R function lme() uses mAIC with an = 1 and SAS Proc Mixed uses both the asymptotic and the finite sa”
Variable Selection in Predictive Regressions (2013, background): “Technical details can be found in Hesterberg, Choi, Meier, and Fraley (2008); Fan and Lv (2010); Belloni and Chernozhukov (2011).”
Asymptotically Honest Confidence Regions for High Dimensional Parameters by the Desparsified Conservative Lasso (2014, background): “For excellent reviews, see Bühlmann and van de Geer (2011), Fan and Lv (2010), and Belloni and Chernozhukov (2011).”
A variational Bayes approach to variable selection (2017, methodology): “Good overviews of the latest approaches to model selection are Johnstone and Titterington (2009); Fan and Lv (2010); M¨uller and Welsh (2010); B¨ulmann and van de Geer (2011); Johnson and Rossell (2012).”
Confidence intervals for low dimensional parameters in high dimensional linear models (2011, methodology): “For further discussion, we refer to related sections in Bühlmann and van de Geer (2011) and recent reviews in Fan and Lv (2010) and Zhang and Zhang (2012).”
GEE analysis of clustered binary data with diverging number of covariates (2011, methodology): “In Fan and Peng (2004), Lam and Fan (2008) and Huang, Horowitz and Ma (2008), the estimator is defined as the minimizer of a certain objective function.”
High-dimensional econometrics and regularized GMM (2018, background): “, [62] for an early paper or [63] for a more recent review.”
Introduction to the LASSO (2018, background): “There is a large literature on variable selection methods for linear models (see [9] and [10]) and for high dimensional problems (see [11] and [12]).”
A flexible semiparametric forecasting model for time series (2015, methodology): “For recent developments and surveys on model averaging and variable selection, the reader is referred to Claeskens and Hjort (2008), Fan and Lv (2008, 2010), Bühlmann and van de Geer (2011) and the references therein.”
Estimation in high-dimensional linear models with deterministic design matrices (2012, background): “If p = e n ν , it is referred to as the ultra-high dimension [Fan and Lv (2010)].”
An Iterative Coordinate Descent Algorithm for High-Dimensional Nonconvex Penalized Quantile Regression (2015, methodology): “The coordinate descent algorithm was systematically investigated for convex problems, such as Lasso, in the independent work of Friedman et al. (2007) and Wu and Lange (2008), the idea of which can be traced back to Fu (1998) and Daubechies, Defrise, and De Mol (2004).”
Variable selection in functional data classification: a maxima-hunting proposal (2013, methodology): “Univariate t-ranking method, denoted by T, is frequently used when selecting relevant variables (see e.g. the review by Fan and Lv (2010)).”
Covariance Estimation: The GLM and Regularization Perspectives (2011, background): “Such an approach will inherit many desirable computational and statistical properties of LASSO and its many improved variants (Efron et al., 2004; Rocha, Zhao and Yu, 2008; Fan and Lv, 2010, Section 3.5).”