Self-Supervised Learning of Pretext-Invariant Representations

By Ishan Misra, L. V. D. van der Maaten. 2019

1,620 citations recorded by the source corpus.

The goal of self-supervised learning from images is to construct image representations that are semantically meaningful via pretext tasks that do not require semantic annotations. Many pretext tasks lead to representations that are covariant with image transformations. We argue that, instead, semantic representations ought to be invariant under such transformations. Specifically, we develop Pretext-Invariant Representation Learning (PIRL, pronounced as `pearl') that learns invariant representations based on pretext tasks. We use PIRL with a commonly used pretext task that involves solving jigsaw puzzles. We find that PIRL substantially improves the semantic quality of the learned image representations. Our approach sets a new state-of-the-art in self-supervised learning from images on several popular benchmarks for self-supervised learning. Despite being unsupervised, PIRL outperforms supervised pre-training in learning image representations for object detection. Altogether, our results demonstrate the potential of self-supervised representations with good invariance properties.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Simple Framework for Contrastive Learning of Visual Representations (2020, positive): “Not only does SimCLR outperform previous work (Figure 1), but it is also simpler, requiring neither specialized architectures (Bachman et al., 2019; Hénaff et al., 2019) nor a memory bank (Wu et al., 2018; Tian et al., 2019; He et al., 2019; Misra & van der Maaten, 2019).”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, background): “Our work builds upon recent success on contrastive learning of visual representations [44, 16, 17, 45, 18, 19, 46, 42, 20, 47, 48, 1], a sub-area within self-supervised learning.”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “Among discriminative methods, contrastive methods [9, 10, 32, 33, 34, 11, 35, 36] currently achieve state-of-the-art performance in self-supervised learning [37, 8, 38, 12].”
VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “Some recent approaches have begun to match or exceed supervised pretraining on ImageNet [22–26], and have been scaled to hundreds of millions [22, 25, 27, 28] or billions [24] of images.”
Improved Baselines with Momentum Contrastive Learning (2020, background): “Recent studies on unsupervised representation learning from images [16, 13, 8, 17, 1, 9, 15, 6, 12, 2] are converging on a central concept known as contrastive learning [5].”
Contrastive Multiview Coding (2019, neutral): “On top of it, strengthening data augmentation with RandAugment [13] yields better or comparable results to other state-of-the-art methods [40,79,87,26,48,18,28,4].”
Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “Recently, methods based on contrastive learning [21, 38] have shown that unsupervised pretraining produces features that surpass the supervised feature representations on many downstream tasks [7, 8, 20, 22, 37].”
High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, neutral): “The latest, most successful, modern family of SSL approaches for images (Misra & Maaten, 2020; Chen et al., 2020; Chen & He, 2020; He et al., 2020; Grill et al., 2020; Caron et al., 2020; 2021; Zbontar et al., 2021; Bardes et al., 2021), have two noteworthy characteristics that markedly distinguish…”
CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, background): “Their storage capacity can be considerably increased by polynomial terms in the energy function (Chen et al., 1986; Psaltis & Cheol, 1986; Baldi & Venkatesh, 1987; Gardner, 1987; Abbott & Arian, 1987; Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov & Hopﬁeld, 2016).”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, background): “In the last two years, various contrastive methods [28, 2, 9, 3, 16, 35, 26, 5, 8, 10, 15, 21, 34, 39] for visual representations have been proposed.”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, background): “This has been particularly successful for natural language processing [43, 45, 9] and is an active research area for computer vision [20, 2, 36, 19, 6].”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, background): “Recently, self-supervised (Chen et al., 2020b; Tian et al., 2020; He et al., 2020; Misra & Maaten, 2020; Li et al., 2021; Grill et al., 2020; Caron et al., 2020) and semi-supervised learning (Yalniz et al., 2019; Xie et al., 2020; Pham et al., 2020) have been studied as alternative paradigms.”
CrossTransformers: spatially-aware few-shot transfer (2020, background): “Some recent works [29, 77] demonstrate that this can improve few-shot learning, although these use self-supervised auxiliary losses rather than integrating self-supervised instance discrimination [15, 24, 55, 81, 94] into episodic training.”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “Unsupervised visual representation learning, or self-supervised learning, aims at obtaining features without using manual annotations and is rapidly closing the performance gap with supervised pretraining in computer vision [9, 20, 37].”
What makes for good views for contrastive learning (2020, methodology): “On top of the “RA-CJ-Blur” augmentations shown in Figure 10, we further reduce the mutual information (or enhance the invariance) of views by using PIRL [46], i.”
Prototypical Contrastive Learning of Unsupervised Representations (2020, methodology): “We follow previous setup [37, 4] and evaluate the performance of such linear classifiers on three datasets, including ImageNet, VOC07, and Places205.”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, neutral): “This paradigm is also being embraced in computer vision (Studer et al. 2019; Misra and Maaten 2020).”