MINE: Mutual Information Neural Estimation

By Ishmael Belghazi, Sai Rajeswar, Aristide Baratin, R. Devon Hjelm, Aaron C. Courville. 2018

747 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Contrastive Multiview Coding (2019, methodology): “We then use the MINE estimator [5] to estimate the mutual information between the views.”
Learning deep representations by mutual information estimation and maximization (2018, methodology): “• We introduce two new measures of representation quality, one based on Mutual Information Neural Estimation (MINE, Belghazi et al., 2018) and a neural dependency measure (NDM) based on the work by Brakel & Bengio (2017), and we use these to bolster our comparison of DIM to different unsupervised…”
Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, methodology): “Furthermore, both IMSAT and DeepINFOMAX compute mutual information over continuous random variables, which requires complex estimators [4], whereas IIC does so for discrete variables with simple and exact computations.”
Learning Robust and Multilingual Speech Representations (2020, neutral): “However, since the exact computation of mutual information is not tractable for continuous variables, recently many estimators have been proposed for enabling unsupervised representation learning with neural networks (Belghazi et al., 2018; van den Oord et al., 2018; Poole et al., 2019).”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, methodology): “For deep neural networks, MINE [4] proposes a method for efficient MI estimation when the encoder is high-dimensional, and has shown promising results for representation learning on images and audio [9, 16].”
What makes for good views for contrastive learning (2020, background): “Alternatively, one may use other MI bounds [5, 45], but we find INCE works well and keep using it.”
NoPeek: Information leakage reduction to share activations in distributed deep learning (2020, background): “Recent approaches to estimate it effectively like [53] are based on”
Variational Information Distillation for Knowledge Transfer (2019, background): “, [4], are both ideas of future interest.”
Learning Representations by Maximizing Mutual Information in Variational Autoencoders (2019, methodology): “A detailed description of this technique can be found in [23].”
Isolating Sources of Disentanglement in VAEs (2018, background): “This metric benefits from advances in efficient computation of mutual information [23] and enforces compactness in addition to disentanglement.”
A Mutual Information Maximization Perspective of Language Representation Learning (2019, methodology): “…1988) and has been the main driver of progress in self-supervised representation learning in other domains such as computer vision, audio processing, and reinforcement learning (Belghazi et al., 2018; van den Oord et al., 2019; Hjelm et al., 2019; Bachman et al., 2019; O’Connor & Veeling, 2019).”
Formal Limitations on the Measurement of Mutual Information (2018, methodology): “Prior to proving the general case, we consider the particular case of the Donsker-Varadhan lower bound on KL divergence (Donsker & Varadhan, 1983; Belghazi et al., 2018).”
LOGAN: Latent Optimisation for Generative Adversarial Networks (2019, background): “Another class of approaches regularises the entropy of the generator outputs to reduce mode collapse (Belghazi et al., 2018; Dieng et al., 2019).”
Near-Optimal Representation Learning for Hierarchical Reinforcement Learning (2018, methodology): “…our theoretical result closely resembles several other recently proposed objectives based on mutual information (van den Oord et al., 2018; Ishmael Belghazi et al., 2018; Hjelm et al., 2018), suggesting an intriguing connection between mutual information and goal representations for hierarchical…”
Imitation Learning via Off-Policy Distribution Matching (2019, background): “For the ﬁrst expectation, previous works have suggested a number of remedies to reduce the bias of mini-batch gradients, such as maintaining moving averages of various quantities (Belghazi et al., 2018).”
Maximum Entropy Generators for Energy-Based Models (2019, methodology): “…is not trivial, and for this purpose we take advantage of very recently proposed GAN-based approaches for maximizing mutual information (Belghazi et al., 2018; Oord et al., 2018; Hjelm et al., 2018), since the mutual information between the input and the output of the generator is…”
Graph Contrastive Learning with Augmentations (2020, background): “Motivated by [46, 47], [24, 21, 48] propose to perform contrastive learning between local and global representations to better capture structure information.”