Maximally Informative Hierarchical Representations of High-Dimensional Data

By Greg Ver Steeg, Aram Galstyan. 2014

70 citations recorded by the source corpus.

We consider a set of probabilistic functions of some input variables as a representation of the inputs. We present bounds on how informative a representation is about input data. We extend these bounds to hierarchical representations so that we can quantify the contribution of each layer towards capturing the information in the original data. The special form of these bounds leads to a simple, bottom-up optimization procedure to construct hierarchical representations that are also maximally informative about the data. This optimization has linear computational complexity and constant sample complexity in the number of variables. These results establish a new approach to unsupervised learning of deep representations that is both principled and practical. We demonstrate the usefulness of the approach on both synthetic and real-world data.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Isolating Sources of Disentanglement in VAEs (2018, methodology): “Each dimension is a normalizing ﬂow of depth 32, and the parameters are trained to maximize the β -TCVAE objective.”
Emergence of Invariance and Disentanglement in Deep Representations (2017, background): ““Disentanglement” is an often-cited property of deep networks (Bengio et al., 2009), but seldom formalized and studied analytically, although Ver Steeg and Galstyan (2015) has suggested studying it using the Total Correlation of the representation, also known as multi-variate mutual information,…”
Estimating Mutual Information for Discrete-Continuous Mixtures (2017, methodology): “MI obeys several appealing properties including the data-processing inequality, invariance under one-to-one transformations and the chain rule [9], which led to a wide use in canonical tasks such as classification [32], clustering [29, 45, 7] and feature selection [2, 12].”
The effectiveness of layer-by-layer training using the information bottleneck principle (2018, methodology): “We note that layer-wise training with a related information-theoretic objective has been studied by Ver Steeg & Galstyan (2015) and Gao et al. (2018), which have shown an effective and practical method to compose and analyze hierarchical representations.”
Auto-Encoding Total Correlation Explanation (2018, methodology): “1 and 2, the following equation holds (Ver Steeg & Galstyan, 2015): The term TC θ ( x ; z ) in Eq.”
The Nearest Neighbor Information Estimator is Adaptively Near Minimax Rate-Optimal (2017, unknown): “They have many applications in modern machine learning tasks, such as classification [48], clustering [46, 58, 10, 41] and feature selection [1, 17].”
Interacting with Predictions: Visual Inspection of Black-box Machine Learning Models (2016, background): “Steeg and Galstyan [45, 41] use total correlation to build a hierarchy of features explaining their role in the data.”
Visual Representations: Defining Properties and Deep Approximations (2014, background): “…from principles and showing that they lead to a particular kind of computational architecture, instead assume a particular architecture and interpret it probabilistically, similarly to Ver Steeg & Galstyan (2015) that uses total correlation as a proxy of information, which is related to our App.”