Information Bottleneck for Gaussian Variables

By Gal Chechik, Amir Globerson, Naftali Tishby, Yair Weiss. 2003

332 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep Variational Information Bottleneck (2017, background): “In our experiments we also train an explicitly deterministic model that has the same form as the stochastic model, except that we just use z = f µe ( x ) as the hidden encoding, and drop the Gaussian layer.”
Wireless Network Intelligence at the Edge (2018, background): “This does not always hold under the miners’ malfunctions that are captured by adding a Gaussian noise N (−0.1, 0.01) to each miner’s aggregate MSIs with the probability of 0.5.”
Understanding disentangling in β-VAE (2018, background): “The β -VAE objective is closely related to the information bottleneck principle [40, 5, 1, 2]: where I ( · ; · ) stands for mutual information and β is a Lagrange multiplier.”
Deep learning and the information bottleneck principle (2015, background): “These bifurcations correspond to phase transitions between different topological representations of ˆ X , such as different cardinality in clustering by deterministic annealing [12], or dimensionality change for continues variables [13].”
Mutual Information Neural Estimation (2018, methodology): “Nonetheless, IB was applied in the case of jointly Gaussian random variables in (Chechik et al., 2005).”
Learning and generalization with the information bottleneck (2008, background): “This framework has several variants and extensions, both to multivariate variables and to continuous representations (see [19, 4] for more details), but these are not the focus of this paper.”
Nonparametric Canonical Correlation Analysis (2015, background): “We next present the algorithms resulting from using one particular choice, namely the kernel density estimates (KDEs) where w ( t ) ∝ e − t/ 2 is the Gaussian kernel, and σ x and σ y are the kernel widths of the two views.”
The effectiveness of layer-by-layer training using the information bottleneck principle (2018, background): “Regularization by injection of noise is common practice in neural net training (Srivastava et al., 2014; Kingma et al., 2015; Wan et al., 2013; Poole et al., 2014), and was also analytically studied for the Gaussian IB case (Chechik et al., 2005).”
Flow of Renyi information in deep neural networks (2016, methodology): “First of all, the IB method works naturally well with discrete bottleneck random variables T , but the inference for its extension to the general solution of continuous random variables T is intractable except for the special case when the source X and relevance Y random variables are jointly Gaussi”
On the information bottleneck theory of deep learning (2018, background): “(2019) 124020 calculation of the mutual information of the representation (without any binning procedure), and, though we do not do so here, direct comparison to the IB bound which is already known for linear Gaussian problems (Chechik et al 2005).”
Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications (2022, unknown): “distribution PS,V , or at least a good estimation of it and that iterating (39) can only be performed for sources with small alphabet (or jointly Gaussian [109], [276], [277]).”
MINE: Mutual Information Neural Estimation (2018, methodology): “Nonetheless, the Information Bottleneck was applied in the case of jointly Gaussian random variables in Chechik et al. (2005).”
A high-bias, low-variance introduction to Machine Learning for physicists (2018, unknown): “B, K-means is a hardassignment limit of the Gaussian mixture model where all cluster variances are assumed to be the same.”
Adversarially Learned Representations for Information Obfuscation and Inference (2019, methodology): “Unlike some other formulations (Chechik & Tishby, 2003; Chechik et al., 2005), we do not make any other assumption on p(x,u, s), i.”
An Information Theoretic Framework for Multi-view Learning (2008, methodology): “When x 1 and x 2 are jointly distributed as a Gaussian, the Gaussian Information Bottleneck method [Chechik et al., 2005] shows that CCA provides an appropriate compression scheme (under the Information Bottleneck criterion [Tishby et al., 1999]).”
Deep multi-view learning methods: A review (2021, methodology): “And indeed, the IB approach also aims to extract a subspaces as CCA in the case of Gaussian variables [107].”