Learning deep representations by mutual information estimation and maximization

By R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Adam Trischler, Yoshua Bengio. 2018

3,036 citations recorded by the source corpus.

This work investigates unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality in the input into the objective can significantly improve a representation’s suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and compares favorably with fully-supervised learning on several classification tasks in with some standard architectures. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation learning objectives for specific end-goals.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “Recent literature has attempted to relate the success of their methods to maximization of mutual information between latent representations (Oord et al., 2018; Hénaff et al., 2019; Hjelm et al., 2018; Bachman et al., 2019).”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, methodology): “The networks fq and fk can be identical [29, 59, 63], partially shared [46, 36, 2], or different [56].”
Shortcut learning in deep neural networks (2020, background): “Extreme dataaugmentation strategies are also the core ingredient of the most successful semi-supervised [124] and self-supervised learning approaches to date [125, 126].”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “Recent methods [67, 25, 4, 45, 24, 60] have advanced the state of the art in terms of learning representations that can linearly separate between the 1000 ImageNet categories [50].”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Other intrinsic scores have been proposed to measure the disentanglement of representations (Locatello et al., 2019), or the mutual information between the input and representations (Hjelm et al., 2018).”
Learning Representations by Maximizing Mutual Information Across Views (2019, methodology): “Our model, which we call Augmented Multiscale DIM (AMDIM), extends the local version of Deep InfoMax introduced by Hjelm et al. [2019] in several ways.”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “For example, camera motion [1, 32], tracking [61], text proximity [41], and other cues [11, 62, 30, 60, 29] can be used to find examples of features that should be ‘similar’ or predictable from one another, whereas negative examples are usually sampled randomly.”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “Recently, methods that maximise the similarity between multiple views (augmented versions) of the same image via contrastive losses [12, 16, 32, 33, 34, 60] stand out due to impressive results on the ImageNet benchmark; we draw inspiration from them (e.”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, background): “Our work builds upon recent success on contrastive learning of visual representations [44, 16, 17, 45, 18, 19, 46, 42, 20, 47, 48, 1], a sub-area within self-supervised learning.”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “Among discriminative methods, contrastive methods [9, 10, 32, 33, 34, 11, 35, 36] currently achieve state-of-the-art performance in self-supervised learning [37, 8, 38, 12].”
Improved Baselines with Momentum Contrastive Learning (2020, background): “1a) [13, 8, 17, 1, 9, 2], the negative keys are from the same batch and updated end-to-end by back-propagation.”
Contrastive Multiview Coding (2019, methodology): “Since we use the patch-based contrastive loss, in the 1 view setting case, CMC coincides with DIM [31].”
An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “Following [45, 21, 2, 9], in MoCo v3 we use the keys that naturally co-exist in the same batch.”
Exploring Simple Siamese Representation Learning (2020, methodology): “This methodology has been recently popularized for un-/self-supervised representation learning [36, 30, 20, 37, 21, 2, 35, 17, 29, 8, 9].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “PIRL is most similar to methods that learn representations that are invariant under standard data augmentation [4, 14, 29, 31, 81, 82].”
On Mutual Information Maximization for Representation Learning (2019, methodology): “Even though the task seems ill-posed as there is no natural objective one should optimize, by leveraging domain knowledge this approach can be successfully applied to a variety of problem areas, including image (Kolesnikov et al., 2019; van den Oord et al., 2018; Hénaff et al., 2019; Tian et al., 20”
Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, background): “Furthermore, both IMSAT and DeepINFOMAX compute mutual information over continuous random variables, which requires complex estimators [4], whereas IIC does so for discrete variables with simple and exact computations.”
Unsupervised Pretraining Transfers Well Across Languages (2020, background): “Contrastive approaches are also related to examplar self-supervision [17, 18, 19].”