Mutual Information Neural Estimation

By Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, R. Devon Hjelm, Aaron C. Courville. 2018

1,674 citations recorded by the source corpus.

We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks. We present a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size, trainable through back-prop, and strongly consistent. We present a handful of applications on which MINE can be used to minimize or maximize mutual information. We apply MINE to improve adversarially trained generative models. We also use MINE to implement Information Bottleneck, applying it to supervised classification; our results demonstrate substantial improvement in flexibility and performance in these settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Representation Learning with Contrastive Predictive Coding (2018, neutral): “InfoNCE is also related to MINE [54].”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, background): “As directly maximizing the mutual information is challenging [37, 3, 50], the InfoNCE loss [36] was proposed to maximize a lower bound of the mutual information I(v1; v2).”
On Mutual Information Maximization for Representation Learning (2019, methodology): “A higher capacity critic should allow for a tighter lower-bound on MI (Belghazi et al., 2018).”
Multi-Task Self-Supervised Learning for Robust Speech Recognition (2020, neutral): “Mutual information is a very meaningful measure of divergence that can capture complex non-linear relationships between random variables [26, 27].”
Excessive Invariance Causes Adversarial Vulnerability (2018, background): “Compared to (Belghazi et al., 2018) there is thus no need for gradient clipping or a switch to the bounded Jensen-Shannon divergence as in (Hjelm et al., 2018) is not necessary.”
On Variational Bounds of Mutual Information (2019, methodology): “Letting a ( y ) be the constant e recovers the bound of Nguyen, Wainwright, and Jordan (Nguyen et al., 2010) also known as f -GAN KL (Nowozin et al., 2016) and MINE-f (Belghazi et al., 2018) 1 : This tractable bound no longer requires learning a ( y ) , but now f ( x, y ) must learn to self-normaliz”
Formal Limitations on the Measurement of Mutual Information (2018, methodology): “ivergence which underlies the approach in MINE. We give an intuitive explanation on why estimating the bound from samples is problematic. We show that the polynomial sample complexity result given by Belghazi et al. (2018) is incorrect. We formally prove that any lower bound on KL divergence cannot ”
InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, methodology): “MINE [51] and InfoNCE [15] compute a lower bound as the MI estimators while CLUB [52] computes an upper bound as the MI estimator.”
Pre-training Molecular Graph Representation with 3D Geometry (2021, neutral): “1 MAXIMIZING MUTUAL INFORMATION Mutual information (MI) measures the non-linear dependence [7] between two random variables: the larger MI, the stronger dependence between the variables.”
Continual Variational Autoencoder Learning via Online Cooperative Memorization (2022, background): “Directly evaluating each Si(j, u) in the high-dimensional data space is intractable since it would require overloaded computations [17] and auxiliary training [7,37].”
Wasserstein Dependency Measure for Representation Learning (2019, background): “A number of self-supervised learning techniques can be shown to be maximizing a lower bound to mutual information between representations of different data modalities [7, 47, 42].”
Learning Intents behind Interactions with Knowledge Graph for Recommendation (2021, methodology): “This module can be simply achieved by applying a statistical measure, such as mutual information [2], Pearson correlation [33], and distance correlation [32, 33, 43], as a regularizer.”
Disentangling Label Distribution for Long-tailed Visual Recognition (2020, methodology): “The usefulness of the DV representation has been broadly shown in the area including mutual information estimation [4, 35, 45] or generative models [4, 44].”
How to Train Your Energy-Based Models (2021, background): “Kumar et al. (2019) propose to estimate the entropy through its connection to mutual information: H ( q φ ( z )) = I ( g φ ( z ) , z ) , which can be estimated from samples with variational lower bounds (Nguyen et al., 2010; Nowozin et al., 2016; Belghazi et al., 2018).”
Unsupervised Learning of Disentangled Speech Content and Style Representation (2020, methodology): “We further disentangle the local and global representations with a mutual information loss [14].”
From Canonical Correlation Analysis to Self-supervised Graph Neural Networks (2021, methodology): “[3] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm.”
The effectiveness of layer-by-layer training using the information bottleneck principle (2018, methodology): “However, as opposed to the original mutual information neural estimator (MINE) of Belghazi et al. (2018), which fails to accurately estimate I(X;Li + ε) in our setting1, we tailor the architecture of the auxiliary net specifically for our case in which Li is a known deterministic function of X .”
Decoupled Contrastive Learning (2021, background): “[17] follows the [3] and extends the idea between the local and global features.”