Supervised feature selection via dependence estimation

By Le Song, Alex Smola, Arthur Gretton, Karsten M. Borgwardt, Justin Bedo. 2007

410 citations recorded by the source corpus.

We introduce a framework for filtering features that employs the Hilbert-Schmidt Independence Criterion (HSIC) as a measure of dependence between the features and the labels. The key idea is that good features should maximise such dependence. Feature selection for various supervised learning problems (including classification and regression) is unified under this framework, and the solutions can be approximated using a backward-elimination algorithm. We demonstrate the usefulness of our method on both artificial and real world datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “The MMD has been applied extensively in many applications, namely, clustering (Jegelka et al. 2009), density estimation (Song et al. 2007a; 2008), (conditional) independence tests (Fukumizu et al. 2008, Doran et al. 2014, Chwialkowski and Gretton 2014), causal discovery (Sgouritsa et al. 2013, Chen…”
Similarity of Neural Network Representations Revisited (2019, background): “They prove that the empirical estimator converges to the population value at a rate of 1/ √ n, and Song et al. (2007) provide an unbiased estimator.”
LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning (2018, methodology): “The dataset was taken from [33] which was constructed from the MNIST data [34].”
Efficient Spectral Feature Selection with Minimum Redundancy (2010, methodology): “There exist many spectral feature selection algorithms: Laplacian score (He, Cai, and Niyogi 2005), Fisher score (Duda, Hart, and Stork 2001), trace ratio (Nie et al. 2008), relief and reliefF (Sikonja and Kononenko 2003), SPEC (Zhao and Liu 2007), and HSIC (Song et al. 2007).”
AM-GCN: Adaptive Multi-channel Graph Convolutional Networks (2020, methodology): “Due to its simplicity and neat theoretical properties, HSIC has been applied to several machine learning tasks [10, 19].”
Sufficient Dimension Reduction via Squared-Loss Mutual Information Estimation (2010, background): “Its usefulness in feature selection scenarios has been shown in [25].”
Feature selection in machine learning: A new perspective (2018, background): “Hilbert-Schmidt dependency criterion [77] shown in formula (9) is a general framework for feature selection, where J (.) measures the dependency of a feature subset to C .”
Discriminative Semi-Supervised Feature Selection Via Manifold Regularization (2009, background): “Following [48], we assume that the number of selected features, i.”
The exact equivalence of distance and kernel methods in hypothesis testing (2018, methodology): “The Hilbert-Schmidt independence criterion is a kernel-based method for testing independence and equally popular in related inference tasks [1, 2, 4, 6, 7, 26, 36].”
Steganalysis by Subtractive Pixel Adjacency Matrix (2009, methodology): “This approach is essentially equal to feature selection using the Hilbert-Schmidt independence criteria with linear kernels [22].”
A Kernel Statistical Test of Independence (2007, methodology): “For the purpose of testing independence, however, we will find it easier to use an alternative, biased empirical estimate [9, Definition 2], obtained by replacing the U-statistics with V-statistics 1 HSIC b (Z) = 1 m 2 m i,j k ij l ij + 1 m 4 m i,j,q,r k ij l qr − 2 1 m 3 m i,j,q k ij l iq = 1 m 2 t”
Supervised principal component analysis: Visualization, classification and regression on subspaces and submanifolds (2011, methodology): “The HSIC has been widely used in many practical applications such as feature selection [25], feature extraction [26], and clustering algorithms [27].”
A Robust-Equitable Measure for Feature Ranking and Selection (2017, methodology): “Filter methods (Kira and Rendell, 1992; Yu and Liu, 2004; Peng et al., 2005; He et al., 2005; Song et al., 2007) pre-select features, without running the learning algorithm.”
A kernel method for unsupervised structured network inference (2009, methodology): “The tool at the core of our kernel method for unsupervised network inference is the Hilbert-Schmidt Independence Criterion (Gretton et al., 2005), and the feature selection method BAHSIC (Song et al., 2007) building on this criterion.”
More generality in efficient multiple kernel learning (2009, methodology): “Empirically, we found the performance of Sparse-SVM to be very similar to that of LP-SVM though, being a QCQP, it took much longer to optimize.”
Feature Selection for Clustering: A Review (2018, background): “According to whether the label information is utilized, different feature selection algorithms can be categorized into supervised [69, 60], unsupervised [18, 46], or semi-supervised algorithms [79, 73].”