The information bottleneck method
By Naftali Tishby, Fernando C Pereira, William Bialek. 2000
4,490 citations recorded by the source corpus.
We define the relevant information in a signal $x\in X$ as being the information that this signal provides about another signal $y\in \Y$. Examples include the information that face images provide about the names of the people portrayed, or the information that speech sounds provide about the words spoken. Understanding the signal $x$ requires more than just predicting $y$, it also requires specifying which features of $\X$ play a role in the prediction. We formalize this problem as that of finding a short code for $\X$ that preserves the maximum information about $\Y$. That is, we squeeze the information that $\X$ provides about $\Y$ through a `bottleneck' formed by a limited set of codewords $\tX$. This constrained optimization problem can be seen as a generalization of rate distortion theory in which the distortion measure $d(x,\x)$ emerges from the joint statistics of $\X$ and $\Y$. This approach yields an exact set of self consistent equations for the coding rules $X \to \tX$ and $\tX \to \Y$. Solutions to these equations can be found by a convergent re-estimation method that generalizes the Blahut-Arimoto algorithm. Our variational principle provides a surprisingly rich framework for discussing a variety of problems in signal processing and learning, as will be described in detail elsewhere.
Subfields: Computer Networks and Communications, Electrical and Electronic Engineering, Management Science and Operations Research, Information Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
- Medhi Bennis
- Merouane Debbah
- Yoshua Bengio
- Jihong Park
- Sergey Levine
- Andrew McCallum
- Daphne Koller
- Stefano Soatto
In Electrical and Electronic Engineering
- Sergey Levine
- Merouane Debbah
- Medhi Bennis
- Yoshua Bengio
- Chelsea Finn
- Ofir Nachum
- Pieter Abbeel
- George Tucker
In Management Science and Operations Research
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ruslan Salakhutdinov
- Ofir Nachum
- Sham M. Kakade
- George Tucker
- Mohammad Norouzi
In Information Theory
- Stefano Soatto
- Alessandro Achille
- Noam Slonim
- Greg Ver Steeg
- Adrian Weller
- Naftali Tishby
- Andrew McCallum
- Shrikanth S. Narayanan
Citations in the corpus
- The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (2019, methodology): “The IB method (Tishby et al., 1999) considers a joint distribution of input-output pairs p ( X, Y ) and aims to extract a compressed representation ˜ X for an input X such that ˜ X retains as much as possible information about the output Y .”
- Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “Choosing an appropriate training objective is an open problem, but a potential guiding principle is that useful representations should make the variability in natural signals more predictable (Tishby et al., 1999; Wiskott & Sejnowski, 2002; Richthofer & Wiskott, 2016).”
- An Unsupervised Autoregressive Model for Speech Representation Learning (2019, background): “In general, however, when a transformation is trained against a certain set of tasks, information independent of the tasks (such as noise or speaker variability, depending on the tasks) tends to be discarded after training [5].”
- What makes for good views for contrastive learning (2020, background): “This idea is related to the idea of minimal sufficient statistics [48] and the Information Bottleneck theory [54, 2], which have been previously articulated in the representation learning literature.”
- Variational Autoencoders for Collaborative Filtering (2018, background): “The authors propose the deep variational information bottleneck, which is a variational approximation to the information bottleneck principle [46].”
- Deep Variational Information Bottleneck (2017, background): “3 This approach is known as the information bottleneck (IB), and was first proposed in Tishby et al. (1999).”
- Wireless Network Intelligence at the Edge (2018, methodology): “(b) IB Under KL Divergence: The formulation (2) in IB is a special case for obtaining the ratedistortion function R(D) with a distortion measure given as a KL divergence [164].”
- Supervised Dimensionality Reduction via Distance Correlation Maximization (2016, neutral): “Given matrices ÊX, ÊY, and column centered matrices X̃, Ỹ, from result of Torgerson (1952) we have that ÊX = −2X̃X̃T and ÊY = −2ỸỸT .”
- Challenges for Transparency (2017, background): “It is possible that a relatively low capacity information bottleneck (as examined by Tishby et al., 1999) will be a useful constraint between agents to develop such structures.”
- Opening the Black Box of Deep Neural Networks via Information (2017, background): “Moreover, they suggested that optimized DNNs layers should approach the Information Bottleneck (IB) bound [Tishby et al. (1999)] of the optimal achievable representations of the input X .”
- Invariant Representations without Adversarial Training (2018, methodology): “We can produce c -invariant codes in the supervised Information Bottleneck context using the relaxation from Eq.”
- Towards a Definition of Disentangled Representations (2018, background): “An alternative to hard wiring inductive biases into the network architecture is to instead learn a representation that is faithful to the underlying data structure [76, 2, 3, 5, 4, 8, 69].”
- InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning (2020, methodology): “Other approaches employing information theoretic principles include Variational GAN (VGAN) [60], which uses an information bottleneck [61] to regularize the discriminator representations; with [62]–[64] extending to minimise divergences apart from the original JS divergence.”
- Where is the Information in a Deep Neural Network? (2019, methodology): “The Information Lagrangian is in the general form of an Information Bottleneck (IB) [17], but is fundamentally different from the IB used in most prior work in deep learning [18], which refers to the activations, rather than the weights.”
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction (2021, methodology): “We explore in this appendix the connection between B AR - LOW T WINS ’ loss function and the Information Bottleneck (IB) principle (Tishby & Zaslavsky, 2015; Tishby et al., 2000).”
- Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning (2015, background): “…problem that forms the basis of a large number of applications in machine learning, including rate distortion theory [4], information bottleneck methods [26], population coding [1], curiosity-driven exploration [24, 19], model selection [3], and intrinsically-motivated reinforcement learning [20].”
- Deep learning and the information bottleneck principle (2015, methodology): “As was shown in [6], the optimal solutions for the IB variational problem satisfy the following self-consistent equations for some value of β , where Z ( x ; β ) is the normalization factor, also known as the partition function.”
- Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations (2017, methodology): “In this context Alemi et al. [2017] build a meaningful representation by using the Information Bottleneck (IB) principle, presented by Tishby et al. [1999].”
- On Variational Bounds of Mutual Information (2019, methodology): “For example in the information bottleneck method (Tishby et al., 2000; Alemi et al., 2016), the representation is optimized to solve a downstream task while being constrained to contain as little information as possible about the input.”