Representation Learning with Contrastive Predictive Coding
By A. van den Oord, Aäron van den Oord, Yazhe Li, Oriol Vinyals. 2018
13,907 citations recorded by the source corpus.
While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. It also makes the model tractable by using negative sampling. While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Sergey Levine
- Quoc V. Le
- Percy Liang
- Ruslan Salakhutdinov
- Luke S. Zettlemoyer
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Andrea Vedaldi
In Representation & Self-Supervised Learning
- Yoshua Bengio
- Trevor Darrell
- Ishan Misra
- Andrea Vedaldi
- Andrew Zisserman
- Dacheng Tao
- Sergey Levine
- Yann LeCun
Citations in the corpus
- Learning Transferable Visual Models From Natural Language Supervision (2021, background): “…in the area of deep metric learning as the multi-class N-pair loss Sohn (2016), was popularized for contrastive representation learning by Oord et al. (2018) as the InfoNCE loss, and was recently adapted for contrastive (text, image) representation learning in the domain of medical…”
- Masked Autoencoders Are Scalable Vision Learners (2021, background): “, [62, 43, 23, 7], which models image similarity and dissimilarity (or only similarity [21, 8]) between two or more views.”
- A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “• CPC v1 and v2 (Oord et al., 2018; Hénaff et al., 2019) define the context prediction task using a deterministic strategy to split examples into patches, and a context aggregation network (a PixelCNN) to aggregate these patches.”
- Evaluating Large Language Models Trained on Code (2021, background): “…al., 2018), computer vision (Van Oord et al., 2016; Menick & Kalchbrenner, 2018; Chen et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal et al., 2020; Baevski et al., 2020), biology (Alley et al., 2019; Rives et al., 2021), and even across multiple…”
- Big Bird: Transformers for Longer Sequences (2020, methodology): “We use two types of regularization in training: • We used a variant of contrastive predictive coding [71] as a dual encoder model.”
- Shortcut learning in deep neural networks (2020, background): “Extreme dataaugmentation strategies are also the core ingredient of the most successful semi-supervised [124] and self-supervised learning approaches to date [125, 126].”
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, background): “Especially in contrastive learning for visual representations [39], remarkable progresses have been made in the past year, for example, SimCLR [24], MoCo [40], and MoCo V2 [41].”
- Are we done with ImageNet? (2020, neutral): “Similarly, successful representation learning techniques applied to ImageNet [9, 10, 11] have yielded corresponding gains elsewhere [12, 13, 14, 15].”
- Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “Recent methods [67, 25, 4, 45, 24, 60] have advanced the state of the art in terms of learning representations that can linearly separate between the 1000 ImageNet categories [50].”
- Learning Representations by Maximizing Mutual Information Across Views (2019, methodology): “Alternatively, one could generate multiple views by repeatedly partitioning the image’s pixels into “past” and “future” sets, with the considered partitions corresponding to a fixed autoregressive ordering, as in Contrastive Predictive Coding [CPC, van den Oord et al., 2018].”
- Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, methodology): “A recent approach for representation learning that has demonstrated strong empirical performance in a variety of modalities is Contrastive Predictive Coding (CPC, [49]).”
- Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, background): “This loss takes the same form as the InfoNCE loss (Oord et al., 2018), and minimizing it leads to encoders that maximally preserve the mutual information between the true pairs under the representation functions.”
- Learning Visual Representations with Caption Annotations (2020, neutral): “ inpainting of missing portions [46], prediction of image rotations [19], spotting artifacts [31], or cross-channel prediction [67]. Besides, recently, contrastive learning-based unsupervised methods [3,26,43,62] have showed signicant improvements. However, computational and data eciency of these me”
- Self-Supervised MultiModal Versatile Networks (2020, methodology): “Given these positive and negative training pairs, we use a contrastive loss [33, 52, 60] to make the positive pairs similar and negative pairs dissimilar in their corresponding joint embedding space.”
- Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, background): “3 As in previous work, we also report performance when training a linear classifier on top of a fixed representation with all labels [31, 16, 17, 1] to directly evaluate SimCLRv2 representations.”
- Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “As BYOL does not use an explicit term to prevent collapse (such as negative examples [10]) while minimizing LBYOL θ,ξ with respect to θ, it may seem that BYOL should converge to a minimum of this loss (5)While we could directly predict the representation y and not a projection z, previous work [8] h”
- VirTex: Learning Visual Representations from Textual Annotations (2020, neutral): “Other approaches use contrastive losses based on context prediction [20, 23], mutual information maximization [21, 53, 54], predicting masked regions [55], and clustering [56–58].”
- Improved Baselines with Momentum Contrastive Learning (2020, background): “1a) [13, 8, 17, 1, 9, 2], the negative keys are from the same batch and updated end-to-end by back-propagation.”