Unsupervised Visual Representation Learning by Context Prediction

By Carl Doersch, Abhinav Kumar Gupta, Alexei A. Efros. 2015

2,985 citations recorded by the source corpus.

This work explores the use of spatial context as a source of free and plentiful supervisory signal for training a rich visual representation. Given only a large, unlabeled image collection, we extract random pairs of patches from each image and train a convolutional neural net to predict the position of the second patch relative to the first. We argue that doing well on this task requires the model to learn to recognize objects and their parts. We demonstrate that the feature representation learned using this within-image context indeed captures visual similarity across images. For example, this representation allows us to perform unsupervised visual discovery of objects like cats, people, and even birds from the Pascal VOC 2011 detection dataset. Furthermore, we show that the learned ConvNet can be used in the R-CNN framework [19] and provides a significant boost over a randomly-initialized ConvNet, resulting in state-of-the-art performance among algorithms which use only Pascal-provided training set annotations.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

An efficient framework for learning sentence representations (2018, methodology): “Tasks such as predicting the relative spatial location of nearby image patches (Doersch et al., 2015), inpainting (Pathak et al., 2016) and solving image jigsaw puzzles (Noroozi & Favaro, 2016) have been successfully used for learning visual feature representations.”
InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, methodology): “A lot of promising recent work originates from the Skip-gram model [14], which inspired the skip-thought vectors [15] and several techniques for unsupervised feature learning of images [16].”
Masked Autoencoders Are Scalable Vision Learners (2021, methodology): “Following [15], we adopt an extra BatchNorm layer [31] without afﬁne transformation ( affine=False ).”
A Simple Framework for Contrastive Learning of Visual Representations (2020, positive): “Although previous work has reported that data augmentation is useful for self-supervised learning (Doersch et al., 2015; Bachman et al., 2019; Hénaff et al., 2019; Asano et al., 2019), we show that data augmentation that does not yield accuracy beneﬁts for supervised learning can still help…”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, background): “Some pretext tasks form pseudo-labels by, e.g ., transformations of a single (“exemplar”) image [17], patch orderings [13, 45], tracking [59] or segmenting objects [47] in videos, or clustering features [3, 4].”
UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “…been applied to many Computer Vision tasks, such as image colorization (Zhang et al., 2016), solving jigsaw puzzles (Noroozi & Favaro, 2016; Trinh et al., 2019), inpainting (Pathak et al., 2016), rotation prediction (Gidaris et al., 2018), and relative location prediction (Doersch et al., 2015).”
Rethinking ImageNet Pre-Training (2018, background): “, for selfsupervised learning [5, 47, 33, 32]), as now we learn that even random initialization could produce excellent results.”
What makes ImageNet good for transfer learning? (2016, background): “Attempted self-supervised tasks include predictions of ego-motion [1, 20], spatial context [11, 33, 30], temporal context [45], and even color [51] and sound [32].”
On Robustness and Transferability of Convolutional Neural Networks (2020, background): “This pre-training is often referred to as representation learning; representations can be trained using supervised [32, 36], weakly-supervised [44], or unsupervised data [13, 14, 66, 70].”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “Following [12, 33], to scale this approach to hundreds of millions of images (frames), we employ a triplet loss [52] encouraging augmentations of the same image to be close and augmentations of different images to be far apart.”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, methodology): “We use a specialized procedure for evaluating patch-based models ( Relative Patch Location (Doersch et al., 2015) and Jigsaw (Noroozi & Favaro, 2016)).”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, background): “There has been substantial recent interest in both vision [38–43] and language around self-supervised representation learning.”
Learning Representations by Maximizing Mutual Information Across Views (2019, background): “E.g., Doersch et al. [2015] and Noroozi and Favaro [2016] learn representations by learning to predict/reconstruct spatial structure.”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “For example, simply asking a network to recognize the spatial layout of an image led to representations that transferred to popular vision tasks such as classification and detection (Doersch et al., 2015; Noroozi & Favaro, 2016).”
S4L: Self-Supervised Semi-Supervised Learning (2019, background): “propose to train a CNN model that predicts relative location of two randomly sampled nonoverlapping image patches [5].”
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era (2017, background): “Our findings also suggest a bright future for unsupervised or self-supervised [10, 43] representation learning approaches.”
Learning Visual Representations with Caption Annotations (2020, background): “tasks consist in predicting missing pieces in manipulated images, for instance context prediction [15], colorization [13,36,66], inpainting of missing portions [46], prediction of image rotations [19], spotting artifacts [31], or cross-channel prediction [67].”
Self-Supervised MultiModal Versatile Networks (2020, background): “single modality), such as predicting the relative position of patches [18, 59], colorization [95], predicting orientation [27] or invariance to transformation [20, 36].”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, background): “There are other approaches to self-supervised learning that are based on handcrafted pretext tasks [52, 31, 53, 54, 27, 55].”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “Most unsupervised methods for representation learning can be categorized as either generative or discriminative [23, 8].”