Data-Efficient Image Recognition with Contrastive Predictive Coding

By Olivier J. H'enaff, A. Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, Aäron van den Oord. 2019

1,574 citations recorded by the source corpus.

Human observers can learn to recognize new categories of images from a handful of examples, yet doing so with artificial ones remains an open challenge. We hypothesize that data-efficient recognition is enabled by representations which make the variability in natural signals more predictable. We therefore revisit and improve Contrastive Predictive Coding, an unsupervised objective for learning such representations. This new implementation produces features which support state-of-the-art linear classification accuracy on the ImageNet dataset. When used as input for non-linear classification with deep neural networks, this representation allows us to use 2-5x less labels than classifiers trained directly on image pixels. Finally, this unsupervised representation substantially improves transfer learning to object detection on the PASCAL VOC dataset, surpassing fully supervised pre-trained ImageNet classifiers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Evaluation & Benchmarks

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, methodology): “We leave exploration of contrastive pre-training (Chen et al., 2020b; He et al., 2020; Bachman et al., 2019; Hénaff et al., 2020) to future work.”
Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Combining CLIP with self-supervision (Henaff, 2020; Chen et al., 2020c) and self-training (Lee; Xie et al.”
A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “• CPC v1 and v2 (Oord et al., 2018; Hénaff et al., 2019) deﬁne the context prediction task using a deterministic strategy to split examples into patches, and a context aggregation network (a PixelCNN) to aggregate these patches.”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, methodology): “patchified inputs [46, 35], carefully tailored receptive fields [2], or combining two networks [56].”
Unsupervised Data Augmentation for Consistency Training (2019, background): “In parallel to our work, S4L [40] and CPC [41] also show significant improvements on ImageNet.”
One Epoch Is All You Need (2019, methodology): “One epoch training and size/iteration adjustment are also applicable to many unsupervised learning algorithms on data of any modality as well as semi-supervised learning as (H´enaff et al., 2019).”
Are we done with ImageNet? (2020, methodology): “We start by assembling a set of 19 models [3, 27, 4, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37] which vary along a number of dimensions, including architecture, training objective and usage of external data (see appendix A).”
Big Transfer (BiT): General Visual Representation Learning (2019, background): “Recent works combine self-supervised and semi-supervised learning to attain good performance with fewer labels on ImageNet [65,17].”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “Recent methods [67, 25, 4, 45, 24, 60] have advanced the state of the art in terms of learning representations that can linearly separate between the 1000 ImageNet categories [50].”
S4L: Self-Supervised Semi-Supervised Learning (2019, methodology): “[12]), we train the S(4)L-Rotation model with a more powerful architecture, such as ResNet152v2 2×wider, and also use large computational budget to tune hyperparameters.”
Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, methodology): “Our work is inspired by the recent line of work on image view-based contrastive learning (Hénaff et al., 2020; Chen et al., 2020a; He et al., 2020; Grill et al., 2020; Sowrirajan et al., 2021; Azizi et al., 2021), but differs from existing studies by the contrastive learning with text modality, whic”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “Given these positive and negative training pairs, we use a contrastive loss [27, 42, 50] to make the positive pairs similar and negative pairs dissimilar in their corresponding joint embedding space.”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “Unsupervised or self-supervised pretraining followed by supervised fine-tuning on a few labeled examples has been extensively used in natural language processing [6, 5, 7–9], but has only shown promising results in computer vision very recently [19, 20, 1].”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “Among discriminative methods, contrastive methods [9, 10, 32, 33, 34, 11, 35, 36] currently achieve state-of-the-art performance in self-supervised learning [37, 8, 38, 12].”
Improved Baselines with Momentum Contrastive Learning (2020, background): “1a) [13, 8, 17, 1, 9, 2], the negative keys are from the same batch and updated end-to-end by back-propagation.”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “In particular, CPC [32, 57] introduces the InfoNCE loss to enforce the model to maximize the conditional probability of some targets (e.”