Ultrahigh dimensional feature selection: beyond the linear model

By Jianqing Fan, Richard J. Samworth, Yichao Wu. Journal of machine learning research, 2009

489 citations recorded by the source corpus.

Variable selection in high-dimensional space characterizes many contemporary problems in scientific discovery and decision making. Many frequently-used techniques are based on independence screening; examples include correlation ranking (Fan and Lv, 2008) or feature selection using a two-sample t-test in high-dimensional classification (Tibshirani et al., 2003). Within the context of the linear model, Fan and Lv (2008) showed that this simple correlation ranking possesses a sure independence screening property under certain conditions and that its revision, called iteratively sure independent screening (ISIS), is needed when the features are marginally unrelated but jointly related to the response variable. In this paper, we extend ISIS, without explicit definition of residuals, to a general pseudo-likelihood framework, which includes generalized linear models as a special case. Even in the least-squares setting, the new method improves ISIS by allowing feature deletion in the iterative process. Our technique allows us to select important features in high-dimensional classification where the popularly used two-sample t-method fails. A new technique is introduced to reduce the false selection rate in the feature screening stage. Several simulated and two real data examples are presented to illustrate the methodology.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/016a3e6a223ffc431a823b428e59bb7657b30a8f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Feature Selection (2016, methodology): “In this case, well-designed feature selection algorithms in linear or sublinear running time are preferred [Fan et al. 2009; Tan et al. 2014].”
Feature selection for high-dimensional class-imbalanced data sets using Support Vector Machines (2014, background): “These problems are (1) any irrelevant feature that is highly correlated with the set of relevant features could be selected and (2) a marginally uncorrelated feature that is jointly correlated with the response might not be selected; see [2,12] for a more general discussion of feature screening.”
Feature Screening via Distance Correlation Learning (2012, methodology): “The aforementioned regularization methods may not perform well for ultrahigh dimensional data due to the simultaneous challenges of computational expediency, statistical accuracy and algorithmic stability (Fan, Samworth and Wu, 2009 ).”
Independent screening for single‐index hazard rate models with ultrahigh dimensional features (2011, methodology): “Following Fan and Lv (2008) and Fan et al. (2009), we would typically choose γ n such that | M n pre | is of order n / log n .”
Variable selection with error control: another look at stability selection (2011, background): “However, the special case of B = 1 of simultaneous selection was studied by Fan et al. (2009), and a variant involving all possible disjoint pairs of subsets was considered in Meinshausen and Bühlmann (2010).”
Stability selection (2008, methodology): “In recent independent work, Fan et al. (2009) made use of a similar condition for error control in regression.”
Bayesian Subset Modeling for High-Dimensional Generalized Linear Models (2013, methodology): “…2013 American Statistical Association Journal of the American Statistical Association June 2013, Vol. 108, No. 502, Theory and Methods DOI: 10.1080/01621459.2012.761942 589 Fan and Lv 2008; Fan and Song 2010) and iterative sure independence screening (ISIS) methods (Fan, Samworth, and Wu 2009).”
Confidence intervals and hypothesis testing for high-dimensional regression (2013, result): “rows, we can compare this with the upper bound proved in Javanmard and Montanari (2013b), cf.”
On Model Parallelization and Scheduling Strategies for Distributed Machine Learning (2014, background): “However, in addition to Big Data, modern large-scale ML problems have started to encounter the so-called Big Model challenge [8, 1, 17], in which models with millions if not billions of parameters and/or variables (such as in deep networks [6] or large-scale topic models [20]) must be estimated from”
High dimensional ordinary least squares projection for screening variables (2015, methodology): “The screening operation has been extended, for example, to generalized linear models (Fan and Fan, 2008; Fan et al., 2009; Fan and Song, 2010), additive models (Fan et al.”
Experimental analysis of new algorithms for learning ternary classifiers (2015, background): “Learning classifiers in very high dimension data has received more and more attention in the past ten years both theoretically [2], [3], [4], [5], [6], [7] and in practice through Data Mining applications [8], [9], [10], [11], [12], [13], [14], [15], [16], [17].”
Challenges of Big Data Analysis (2013, unknown): “Besides large-scale optimization algorithms, Big Data also motivate the development of majorization–minimization algorithms [25–27], ‘large-scale screening and small-scale optimization’ framework [28], parallel computing methods [29– 31] and approximate algorithms that are scalable to large sample s”