Unsupervised Representation Learning by Predicting Image Rotations

By Spyros Gidaris, Praveer Singh, Nikos Komodakis. 2018

3,609 citations recorded by the source corpus.

Over the last years, deep convolutional neural networks (ConvNets) have transformed the field of computer vision thanks to their unparalleled capacity to learn high level semantic image features. However, in order to successfully learn those features, they usually require massive amounts of manually labeled data, which is both expensive and impractical to scale. Therefore, unsupervised semantic feature learning, i.e., learning without requiring manual annotation effort, is of crucial importance in order to successfully harvest the vast amount of visual data that are available today. In our work we propose to learn image features by training ConvNets to recognize the 2d rotation that is applied to the image that it gets as input. We demonstrate both qualitatively and quantitatively that this apparently simple task actually provides a very powerful supervisory signal for semantic feature learning. We exhaustively evaluate our method in various unsupervised feature learning benchmarks and we exhibit in all of them state-of-the-art performance. Specifically, our results on those benchmarks demonstrate dramatic improvements w.r.t. prior state-of-the-art approaches in unsupervised representation learning and thus significantly close the gap with supervised feature learning. For instance, in PASCAL VOC 2007 detection task our unsupervised pre-trained AlexNet model achieves the state-of-the-art (among unsupervised methods) mAP of 54.4% that is only 2.4 points lower from the supervised case. We get similarly striking results when we transfer our unsupervised learned features on various other tasks, such as ImageNet classification, PASCAL classification, PASCAL segmentation, and CIFAR-10 classification. The code and models of our paper will be published on: this https URL .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Masked Autoencoders Are Scalable Vision Learners (2021, background): “Self-supervised learning approaches have seen signiﬁcant interest in computer vision, often focusing on different pre-text tasks for pre-training [15, 61, 42, 70, 45, 17].”
A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “Many such approaches have relied on heuristics to design pretext tasks (Doersch et al., 2015; Zhang et al., 2016; Noroozi & Favaro, 2016; Gidaris et al., 2018), which could limit the generality of the learned representations.”
UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “…been applied to many Computer Vision tasks, such as image colorization (Zhang et al., 2016), solving jigsaw puzzles (Noroozi & Favaro, 2016; Trinh et al., 2019), inpainting (Pathak et al., 2016), rotation prediction (Gidaris et al., 2018), and relative location prediction (Doersch et al., 2015).”
On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “While there exist many approaches on how to transfer to the new task, the most common approach in modern deep learning, which we use, is to (i) train a model on Ppre−train (using perhaps an auxiliary, self-supervised task [15, 22]), and then (ii) train a model on Ptrain by initializing the model wei”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “To define the frame/shot-level loss LS, we propose to build on any SSL loss designed for images, such as classifying exemplars [15], solving jigsaw puzzles of image patches [43], or rotation prediction [18].”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, methodology): “The image-based models include ROTATION (Gidaris et al., 2018) and E XEM - PLAR (Dosovitskiy et al., 2014).”
Learning Representations by Maximizing Mutual Information Across Views (2019, methodology): “→ The methods [1, 2, 3, 4, 5] are from: [Gidaris et al., 2018, Dosovitskiy et al., 2014, Doersch and Zisserman, 2017, Noroozi and Favaro, 2016, van den Oord et al., 2018].”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “, 2017) and image orientation (Gidaris et al., 2018), and invariance to data augmentation (Dosovitskiy et al.”
S4L: Self-Supervised Semi-Supervised Learning (2019, methodology): “We demonstrate our framework on two prominent selfsupervised techniques: predicting image rotation [9] and exemplar [6].”
Learning Visual Representations with Caption Annotations (2020, methodology): “For reference, we also provide results for two self-supervised approaches: RotNet [19] and DeeperCluster [7].”
Self-Supervised MultiModal Versatile Networks (2020, background): “single modality), such as predicting the relative position of patches [18, 59], colorization [95], predicting orientation [27] or invariance to transformation [20, 36].”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, background): “There are other approaches to self-supervised learning that are based on handcrafted pretext tasks [52, 31, 53, 54, 27, 55].”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “In particular, relative patch prediction [23, 40], colorizing grayscale images [41, 42], image inpainting [43], image jigsaw puzzle [44], image super-resolution [45], and geometric transformations [46, 47] have been shown to be useful.”
Contrastive Multiview Coding (2019, methodology): “As many previous unsupervised methods are evaluated with AlexNet [43] on ImageNet [16, 42, 17, 84, 54, 18, 85, 55, 22, 11, 83], we also include the the results of CMC using this network.”
BEiT: BERT Pre-Training of Image Transformers (2021, neutral): “, 2016), and predicting rotation angles (Komodakis and Gidaris, 2018).”
Vector-quantized Image Modeling with Improved VQGAN (2021, background): “However, their performance is usually not superior to competing self-supervised approaches that solve auxiliary classification tasks (Noroozi & Favaro, 2016a; Gidaris et al., 2018a; van den Oord et al., 2018).”
High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, neutral): “Pretext tasks for the image domain include solving Jigsaw-puzzles (Noroozi & Favaro, 2016), predicting rotations or affine transformations (Gidaris et al., 2018; Zhang et al., 2019b) or discriminating instances (Wu et al., 2018; van den Oord et al., 2018).”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “Our motivation for this design of g(vIt) is the desire to remain as close as possible to the covariant pretext task of [20, 21, 54].”