Understanding the Limitations of Variational Mutual Information Estimators
By Jiaming Song, Stefano Ermon. 2019
248 citations recorded by the source corpus.
Variational approaches based on neural networks are showing promise for estimating mutual information (MI) between high dimensional variables. However, they can be difficult to use in practice due to poorly understood bias/variance tradeoffs. We theoretically show that, under some conditions, estimators such as MINE exhibit variance that could grow exponentially with the true amount of underlying MI. We also empirically demonstrate that existing estimators fail to satisfy basic self-consistency properties of MI, such as data processing and additivity under independence. Based on a unified perspective of variational approaches, we develop a new estimator that focuses on variance reduction. Empirical results on standard benchmark tasks demonstrate that our proposed estimator exhibits improved bias-variance trade-offs on standard benchmark tasks.
Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods, Information Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Ruslan Salakhutdinov
- Percy Liang
- Jianfeng Gao
- Yoshua Bengio
- Sergey Levine
- Stefano Ermon
- Zhe Gan
In Probabilistic & Bayesian Methods
In Information Theory
- Stefano Ermon
- Jiaming Song
- Zhe Gan
- Lawrence Carin
- Ruslan Salakhutdinov
- Yao-Hung Hubert Tsai
- Honglak Lee
- Polina Golland
Citations in the corpus
- Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, background): “As directly maximizing the mutual information is challenging [37, 3, 50], the InfoNCE loss [36] was proposed to maximize a lower bound of the mutual information I(v1; v2).”
- A Theory of Usable Information Under Computational Constraints (2020, background): “…the CPC estimator suffers from large bias, since its estimates saturate at log N where N is the batch size (van den Oord et al., 2018; Poole et al., 2019); the NWJ estimator suffers from large variance that grows at least exponentially in the ground-truth mutual information (Song & Ermon, 2019).”
- Disentangling Label Distribution for Long-tailed Visual Recognition (2020, methodology): “However, [54, 12, 41] have pointed out the instability of directly using the DV representation.”
- Graph Clustering with Graph Neural Networks (2020, background): “Unsupervised training of GNNs is commonly done via maximizing mutual information [3, 26, 53, 57] in a self-supervised fashion.”
- Negative Data Augmentation (2021, background): “As n → ∞ , the optimal g φ corresponds to an un-normalized density ratio between the joint distribution and the product of marginals, and the CPC objective matches its upper bound which is the mutual information between X and Z (Poole et al., 2019a; Song & Ermon, 2019).”
- Contrastive Learning for Weakly Supervised Phrase Grounding (2020, methodology): “t generated the samples. Thus, recent eorts rely on variational estimation of MI [3,20,6,30]. An overview of such estimators is discussed in [31,40] while the statistical limitations are reviewed in [26,34]. In practice, MI-based representation learning models are often trained by maximizing an esti”
- Multimodal Representation Learning via Maximization of Local Mutual Information (2021, methodology): “We leverage the recent advances that employ neural network discriminators for MI estimation and maximization [2, 18, 24, 27].”
- Self-supervised Learning from a Multi-view Perspective (2020, background): “…suggest contrastive predictive coding (CPC) (Oord et al., 2018) 4 , which is a mutual information lower bound with low variance (Poole et al., 2019; Song & Ermon, 2019): where F S : S → Z is a deterministic mapping and G is a project head that projects a representation in Z into a…”
- Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE (2021, background): “The prevailing conjecture is that these biased contrastive learners ben-efit from a lower estimation variance [21, 32].”