Variational Information Maximization for Feature Selection
By Shuyang Gao, Greg Ver Steeg, Aram Galstyan. 2016
53 citations recorded by the source corpus.
Feature selection is one of the most fundamental problems in machine learning. An extensive body of work on information-theoretic feature selection exists which is based on maximizing mutual information between subsets of features and class labels. Practical methods are forced to rely on approximations due to the difficulty of estimating mutual information. We demonstrate that approximations made by existing methods are based on unrealistic assumptions. We formulate a more flexible and general class of assumptions based on variational distributions and use them to tractably generate lower bounds for mutual information. These bounds define a novel information-theoretic framework for feature selection, which we prove to be optimal under tree graphical models with proper choice of variational distributions. Our experiments demonstrate that the proposed method strongly outperforms existing information-theoretic feature selection approaches.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jiliang Tang
- Le Song
- Michael I. Jordan
- Huan Liu
- Arthur Gretton
- Bernhard Schölkopf
- Jundong Li
- Tatsuya Harada
In Computer Vision and Pattern Recognition
- Tatsuya Harada
- Luc Van Gool
- Yi Yang
- Dacheng Tao
- Jiliang Tang
- Andrea Vedaldi
- Ming-Hsuan Yang
- Trevor Darrell
Citations in the corpus
- Learning to Explain With Complemental Examples (2018, methodology): “Similar to the method in [9], we used the following”
- Generative causal explanations of black-box classifiers (2020, methodology): “It is interesting to note that this metric, derived in our work from first principles of causality using the DAG in Figure 1(b), has been used as a heuristic for selecting informative features in other work on explanation [56, 11, 40, 57–59].”
- Learning to Explain: An Information-Theoretic Perspective on Model Interpretation (2018, background): “The mutual information has been a popular criteria in feature selection, where one selects the subset of features that approximately maximizes the mutual information between the response variable and the selected features (Gao et al., 2016; Peng et al., 2005).”
- Feature Selection (2016, background): “…(Koller and Sahami 1995; Guyon and Elisseeff 2003), mutual information (Yu and Liu 2003; Peng et al. 2005; Nguyen et al. 2014; Shishkin et al. 2016; Gao et al. 2016), feature ability to preserve data manifold structure (He et al. 2005; Zhao and Liu 2007; Gu et al. 2011b; Jiang and Ren 2011), and…”