On Variational Bounds of Mutual Information

By Ben Poole, Sherjil Ozair, Aäron van den Oord, Alexander A. Alemi, George Tucker. 2019

1,036 citations recorded by the source corpus.

Estimating and optimizing Mutual Information (MI) is core to many problems in machine learning; however, bounding MI in high dimensions is challenging. To establish tractable and scalable objectives, recent work has turned to variational bounds parameterized by neural networks, but the relationships and tradeoffs between these bounds remains unclear. In this work, we unify these recent developments in a single framework. We find that the existing variational lower bounds degrade when the MI is large, exhibiting either high bias or high variance. To address this problem, we introduce a continuum of lower bounds that encompasses previous bounds and flexibly trades off bias and variance. On high-dimensional, controlled problems, we empirically characterize the bias and variance of the bounds and their gradients and demonstrate the effectiveness of our new bounds for estimation and representation learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “Let us consider the following objective that extends the InfoNCE objective [10, 84] (see Appendix G.”
Contrastive Multiview Coding (2019, background): “A formal proof is given by [57, 62], which we recapitulate in supplement, showing that:”
Dual Contradistinctive Generative Autoencoder (2020, positive): “Remark 1 (From [49, 54]) The following objective is minimized, i.e., the optimal critic h is achieved, when h ( f θ ( z ) , x ) = log p ( x | z ) + c ( x ) where c ( x ) is any function that does not depend on z .”
CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, methodology): “The “InfoLOOB” objective is called “Leave one out upper bound” in Poole et al. (2019) and “L1Out” in Cheng et al. (2020).”
On Mutual Information Maximization for Representation Learning (2019, methodology): “In fact, while InfoNCE is a lower bound on MI if the negative samples are drawn from the true marginal distribution (Poole et al., 2019), i.”
Learning Robust and Multilingual Speech Representations (2020, neutral): “However, since the exact computation of mutual information is not tractable for continuous variables, recently many estimators have been proposed for enabling unsupervised representation learning with neural networks (Belghazi et al., 2018; van den Oord et al., 2018; Poole et al., 2019).”
What makes for good views for contrastive learning (2020, background): “Alternatively, one may use other MI bounds [5, 45], but we find INCE works well and keep using it.”
On the Fairness of Disentangled Representations (2019, background): “State-of-the-art approaches for unsupervised disentanglement learning are based on representations learned by VAEs [51].”
Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, background): “However, the computation of MI values is intractable [44,18], since the form of joint probability distribution p(a,b) in Eq.”
Formal Limitations on the Measurement of Mutual Information (2018, background): “Poole et al. (2019) discuss the large variance of these estimators.”
Dream to Control: Learning Behaviors by Latent Imagination (2019, methodology): “…we subtract the constant marginal probability of the data under the variational encoder, apply Bayes rule, and use the InfoNCE mini-batch bound (Poole et al., 2019), For the second term, we use the non-negativity of the KL divergence to obtain an upper bound, This lower bounds the objective.”
Multi-label Contrastive Predictive Coding (2020, methodology): “A variety of mutual information estimators with different bias-variance trade-offs have been proposed for representation learning [39, 48, 3, 40].”
Self-supervised Learning from a Multi-view Perspective (2020, background): “…objective, we suggest contrastive predictive coding (CPC) (Oord et al., 2018) 4 , which is a mutual information lower bound with low variance (Poole et al., 2019; Song & Ermon, 2019): where F S : S → Z is a deterministic mapping and G is a project head that projects a representation in Z…”
Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis (2021, background): “ficacious in removing redundant information irrelevant to the downstream task and capturing invariant trends or messages across time or different domains (Poole et al., 2019), and has been shown remarkable success in the field of representation learning (Hjelm et al.”
CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information (2020, methodology): “With N sample pairs { ( x i , y i ) } Ni =1 , Poole et al. (2019) replaces r ( y ) with a Monte Carlo approximation r i ( y ) = 1 N − 1 j (cid:54) = i p ( y | x j ) ≈ p ( y ) and derives a leave-one-out upper bound (L 1 Out): This bound does not require any additional parameters, but highly depends…”
Understanding the Limitations of Variational Mutual Information Estimators (2019, methodology): “Recently, there has been a surge of interest in MI estimation with variational approaches (Barber & Agakov, 2003; Nguyen et al., 2010; Donsker & Varadhan, 1975), which can be naturally combined with deep learning methods (Alemi et al., 2016; van den Oord et al., 2018; Poole et al., 2019).”
Learning Video Representations using Contrastive Bidirectional Transformer (2019, neutral): “, (Oord et al., 2018; Poole et al., 2019)).”