Learning Representations by Maximizing Mutual Information Across Views

By Philip Bachman, R. Devon Hjelm, William Buchwalter. 2019

1,654 citations recorded by the source corpus.

We propose an approach to self-supervised representation learning based on maximizing mutual information between features extracted from multiple views of a shared context. For example, one could produce multiple views of a local spatio-temporal context by observing it from different locations (e.g., camera positions within a scene), and via different modalities (e.g., tactile, auditory, or visual). Or, an ImageNet image could provide a context from which one produces multiple views by repeatedly applying data augmentation. Maximizing mutual information between features extracted from these views requires capturing information about high-level factors whose influence spans multiple views -- e.g., presence of certain objects or occurrence of certain events. Following our proposed approach, we develop a model which learns image representations that significantly outperform prior methods on the tasks we consider. Most notably, using self-supervised learning, our model learns representations which achieve 68.1% accuracy on ImageNet using standard linear evaluation. This beats prior results by over 12% and concurrent results by 7%. When we extend our model to use mixture-based representations, segmentation behaviour emerges as a natural side-effect. Our code is available online: this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “We leave exploration of contrastive pre-training (Chen et al., 2020b; He et al., 2020; Bachman et al., 2019; H ´ enaff et al., 2020) to future work.”
Learning Transferable Visual Models From Natural Language Supervision (2021, neutral): “We remove the non-linear projection between the representation and the contrastive embedding space, a change which was introduced by Bachman et al. (2019) and popularized by Chen et al. (2020b).”
A Simple Framework for Contrastive Learning of Visual Representations (2020, positive): “The best self-supervised model that reports linear evaluation result on CIFAR-10 is AMDIM (Bachman et al., 2019), which achieves 91.2% with a model 25 × larger than ours.”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, methodology): “Following [63, 2], we take two random “views” of the same image under random data augmentation to form a positive pair.”
Experience Grounds Language (2020, background): “Alternative, unsupervised training regimes (Bachman et al., 2019) also open up research towards automatic concept formation.”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “Recent methods [67, 25, 4, 45, 24, 60] have advanced the state of the art in terms of learning representations that can linearly separate between the 1000 ImageNet categories [50].”
Learning Visual Representations with Caption Annotations (2020, background): “Besides, recently, contrastive learning-based unsupervised methods [3,26,43,62] have showed significant improvements.”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “Recently, methods that maximise the similarity between multiple views (augmented versions) of the same image via contrastive losses [12, 16, 32, 33, 34, 60] stand out due to impressive results on the ImageNet benchmark; we draw inspiration from them (e.”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, background): “Our work builds upon recent success on contrastive learning of visual representations [44, 16, 17, 45, 18, 19, 46, 42, 20, 47, 48, 1], a sub-area within self-supervised learning.”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “Among discriminative methods, contrastive methods [9, 10, 32, 33, 34, 11, 35, 36] currently achieve state-of-the-art performance in self-supervised learning [37, 8, 38, 12].”
VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “Other approaches use contrastive losses for context prediction [20, 23], mutual information maximization [21, 53, 54], predicting masked regions [55], and clustering [56].”
Improved Baselines with Momentum Contrastive Learning (2020, background): “1a) [13, 8, 17, 1, 9, 2], the negative keys are from the same batch and updated end-to-end by back-propagation.”
Contrastive Multiview Coding (2019, methodology): “A preliminary exploration of this is given in Section 4.1.2, where we achieve 60.1% top-1 accuracy on ImageNet with CMC using Resnet-101, compared to 61.0% with CPC++ using Resnet-170 and 60.2% with Deep Infomax++ using a heavily-customized Resnet.”
An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “Following [45, 21, 2, 9], in MoCo v3 we use the keys that naturally co-exist in the same batch.”
Exploring Simple Siamese Representation Learning (2020, methodology): “Simple and effective instantiations of contrastive learning have been developed using Siamese networks [37, 2, 17, 8, 9].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “4%, which is close to the accuracy obtained by AMDIM [4] with a model that has 6× more parameters.”
On Mutual Information Maximization for Representation Learning (2019, methodology): “…While promising results in other domains have been presented in the literature, we will focus on unsupervised image representation learning techniques that have achieved state-of-the-art performance on image classiﬁcation tasks (Hénaff et al., 2019; Tian et al., 2019; Bachman et al., 2019).”