Contrastive Multiview Coding

By Yonglong Tian, Dilip Krishnan, Phillip Isola. 2019

2,769 citations recorded by the source corpus.

Humans view the world through many sensory channels, e.g., the long-wavelength light channel, viewed by the left eye, or the high-frequency vibrations channel, heard by the right ear. Each view is noisy and incomplete, but important factors, such as physics, geometry, and semantics, tend to be shared between all views (e.g., a "dog" can be seen, heard, and felt). We investigate the classic hypothesis that a powerful representation is one that models view-invariant factors. We study this hypothesis under the framework of multiview contrastive learning, where we learn a representation that aims to maximize mutual information between different views of the same scene but is otherwise compact. Our approach scales to any number of views, and is view-agnostic. We analyze key properties of the approach that make it work, finding that the contrastive loss outperforms a popular alternative based on cross-view prediction, and that the more views we learn from, the better the resulting representation captures underlying scene semantics. Our approach achieves state-of-the-art results on image and video unsupervised learning benchmarks. Code is released at: this http URL.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Recent work in contrastive representation learning for images has found that contrastive objectives can learn better representations than their equivalent predictive objective (Tian et al., 2019).”
A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “Table 6 compares our results with previous approaches (Zhuang et al., 2019; He et al., 2019a; Misra & van der Maaten, 2019; Hénaff et al., 2019; Kolesnikov et al., 2019; Donahue & Simonyan, 2019; Bachman et al., 2019; Tian et al., 2019) in the linear evaluation setting.”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, methodology): “The pretext task in contrastive predictive coding (CPC) [46] is a form of context auto-encoding [48], and in contrastive multiview coding (CMC) [56] it is related to colorization [64].”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “. The latter two require distilling the temporal information from video pixels into optical-ﬂow ﬁelds. Optical-ﬂow has been treated as a separate modality from the RGB pixels in a multi-modal setting [48,56]. Even beyond optical-ﬂow, videos, as found on the Internet, are inherently multi-modal, as t”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “Contrastive Multiview Coding (Tian et al., 2019) maximizes the mutual information between representations of different views of the same observation.”
Self-Supervised MultiModal Versatile Networks (2020, background): “In contrast, we argue that using multiple modalities as different views is simpler and more natural [80].”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, background): “Our work builds upon recent success on contrastive learning of visual representations [44, 16, 17, 45, 18, 19, 46, 42, 20, 47, 48, 1], a sub-area within self-supervised learning.”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “Among them, state-of-the-art contrastive methods [8, 9, 10, 11, 12] are trained by reducing the distance between representations of different augmented views of the same image (‘positive pairs’), and increasing the distance between representations of augmented views from different images (‘negative ”
VirTex: Learning Visual Representations from Textual Annotations (2020, neutral): “Other approaches use contrastive losses based on context prediction [20, 23], mutual information maximization [21, 53, 54], predicting masked regions [55], and clustering [56–58].”
Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, methodology): “We follow the training recipe in CMC [49] and MoCo [16] except for two differences.”
Improved Baselines with Momentum Contrastive Learning (2020, background): “s. We hope this will make state-of-the-art unsupervised learning research more accessible. Code will be made public. 1. Introduction Recent studies on unsupervised representation learning from images [16,13,8,17,1,9,15,6,12,2] are converging on a central concept known as contrastive learning [5]. Th”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, background): “Some of these tasks include: temporal ordering of video clips or frames [24, 46, 54, 86], predicting geometric transformations [38], maximizing the mutual information of multiple views [74], predicting motion and appearance [78], predicting the future, the past or a portion of masked input in the fe”
Exploring Simple Siamese Representation Learning (2020, background): “contrastive learning [16] is to attract the positive sample pairs and repulse the negative sample pairs. This methodology has been recently popularized for un-/self-supervised representation learning [36,30,20,37,21,2,35,17,29,8,9]. Simple and effective instantiations of contrastive learning have be”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, background): “In the last two years, various contrastive methods [2, 3, 5, 8, 9, 10, 15, 16, 21, 26, 28, 34, 35, 39] for visual representations have been proposed.”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “For example, CMC [73] uses a combination of two ResNet-50 models and trains the linear classifier for longer to obtain 64.1% accuracy.”
On Mutual Information Maximization for Representation Learning (2019, methodology): “Following the widely adopted downstream linear evaluation protocol (Kolesnikov et al., 2019; van den Oord et al., 2018; Hénaff et al., 2019; Tian et al., 2019; Hjelm et al., 2019; Bachman et al., 2019), we train a linear classiﬁer 3 for digit classiﬁcation on the learned representation using all…”
Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, methodology): “Several concurrent methods [67, 2, 28] optimizing mutual information of representations from different views of the same image achieve comparable performances to LA while using significantly larger architectures.”