Multivariate Information Bottleneck

By Nir Friedman, Ori Mosenzon, Noam Slonim, Naftali Tishby. 2001

227 citations recorded by the source corpus.

The information bottleneck (IB) method is an unsupervised model independent data organization technique. Given a joint distribution, p(X, Y), this method constructs a new variable, T, that extracts partitions, or clusters, over the values of X that are informative about Y. Algorithms that are motivated by the IB method have already been applied to text classification, gene expression, neural code, and spectral analysis. Here, we introduce a general principled framework for multivariate extensions of the IB method. This allows us to consider multiple systems of data partitions that are interrelated. Our approach utilizes Bayesian networks for specifying the systems of clusters and which information terms should be maintained. We show that this construction provides insights about bottleneck variations and enables us to characterize the solutions of these variations. We also present four different algorithmic approaches that allow us to construct solutions in practice and apply them to several real-world problems.

Subfields: Artificial Intelligence, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

Citations in the corpus

Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, background): “More generally, learning from paired data has been explored in co-clustering [25, 16] and in other works [50] that build on the information bottleneck principle [20].”
Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “Interestingly, we can exploit the following result from in Friedman et al. (2001) to setup the objective for π , without explicitly knowing η : where I P and I N represents the multi-information for distribution P on the respective graphical models.”
Information Bottleneck for Gaussian Variables (2003, methodology): “…useful for a range of applications from documents clustering (Slonim and Tishby, 2000) through neural code analysis (Dimitrov and Miller, 2001) to gene expression analysis (Friedman et al., 2001; Sinkkonen and Kaski, 2001) (for a more detailed review of IB clustering algorithms see Slonim (2003)).”
Learning and generalization with the information bottleneck (2008, methodology): “The method has proven to be useful for a number of important applications (see [24, 8, 21] and references therein), but its learning theoretic justification has”
The Information Bottleneck : Theory and Applications (2006, methodology): “…Bottleneck: Theory and Applications Thesis submitted for the degree “Doctor of Philosophy” Noam Slonim Submitted to the Senate of the Hebrew University in the year 2002 This work was carried out under the supervision of Prof. Naftali Tishby This thesis is dedicated to the memory of my father”
Invariant Information Distillation for Unsupervised Image Segmentation and Clustering (2018, background): “The multivariate information bottleneck (MIB) Friedman et al. [2001], Wang et al. [2010] version of Eq.”
Information Bottleneck Domain Adaptation with Privileged Information for Visual Recognition (2016, methodology): “The multivariate IB method [41] implements this idea by using the multi-information of X, which is the information shared by X1, · · · , Xn, i.”
Anchored Correlation Explanation: Topic Modeling with Minimal Domain Knowledge (2016, background): “The information bottleneck formulates a trade-off between compressing data X into a representation Y , and preserving the information in X that is relevant to Z (typically labels in a supervised learning task) (Tishby et al., 2000; Friedman et al., 2001).”
Information Bottleneck Learning Using Privileged Information for Visual Recognition (2016, methodology): “The multi-information I with respect to a Bayesian networkG defined over X ∼ p(X) is computed as in [34], i.”
Maximally Informative Hierarchical Representations of High-Dimensional Data (2014, unknown): “Al-ternately, we could estimate these density ratios non-parametrically [17] or using other prior knowledge.”
Information Bottleneck Co-clustering (2010, methodology): “The statistics of the data to be preserved can be captured by properly defining conditional expectation-based constraints.”
Disambiguating Web appearances of people in a social network (2005, methodology): “[8] propose the Multivariate Information Bottleneck (MIB) framework: they consider clustering instances of a set of variables X = (X1, .”