Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
By Mehdi Noroozi, Paolo Favaro. 2016
3,276 citations recorded by the source corpus.
We propose a novel unsupervised learning approach to build features suitable for object detection and classification. The features are pre-trained on a large dataset without human annotation and later transferred via fine-tuning on a different, smaller and labeled dataset. The pre-training consists of solving jigsaw puzzles of natural images. To facilitate the transfer of features to other tasks, we introduce the context-free network (CFN), a siamese-ennead convolutional neural network. The features correspond to the columns of the CFN and they process image tiles independently (i.e., free of context). The later layers of the CFN then use the features to identify their geometric arrangement. Our experimental evaluations show that the learned features capture semantically relevant content. We pre-train the CFN on the training set of the ILSVRC2012 dataset and transfer the features on the combined training and validation set of Pascal VOC 2007 for object detection (via fast RCNN) and classification. These features outperform all current unsupervised features with \(51.8\,\%\) for detection and \(68.6\,\%\) for classification, and reduce the gap with supervised learning (\(56.5\,\%\) and \(78.2\,\%\) respectively).
Subfields: Computer Vision and Pattern Recognition, Geology, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Andrea Vedaldi
In Geology
- Jitendra Malik
- Ishan Misra
- Armand Joulin
- Leonidas J. Guibas
- Philipp Krahenbuhl
- Rohit Girdhar
- Andrea Vedaldi
- Kristen Grauman
In Representation & Self-Supervised Learning
- Ishan Misra
- Andrea Vedaldi
- Andrew Zisserman
- Trevor Darrell
- Dacheng Tao
- Armand Joulin
- Yann LeCun
- Kaiming He
Citations in the corpus
- An efficient framework for learning sentence representations (2018, methodology): “Tasks such as predicting the relative spatial location of nearby image patches (Doersch et al., 2015), inpainting (Pathak et al., 2016) and solving image jigsaw puzzles (Noroozi & Favaro, 2016) have been successfully used for learning visual feature representations.”
- Masked Autoencoders Are Scalable Vision Learners (2021, background): “Self-supervised learning approaches have seen significant interest in computer vision, often focusing on different pretext tasks for pre-training [15, 50, 35, 59, 38, 17].”
- A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “The recent renaissance of self-supervised learning began with artificially designed pre-text tasks, such as relative patch prediction (Doersch et al., 2015), solving jigsaw puzzles (Noroozi & Favaro, 2016), colorization (Zhang et al., 2016) and rotation prediction (Gi-daris et al., 2018).”
- Momentum Contrast for Unsupervised Visual Representation Learning (2019, methodology): “The Jigsaw [45] result is from the ResNet-based implementation in [26].”
- UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “…of supervision, which has been applied to many Computer Vision tasks, such as image colorization (Zhang et al., 2016), solving jigsaw puzzles (Noroozi & Favaro, 2016; Trinh et al., 2019), inpainting (Pathak et al., 2016), rotation prediction (Gidaris et al., 2018), and relative location…”
- What makes ImageNet good for transfer learning? (2016, background): “Attempted self-supervised tasks include predictions of ego-motion [1, 20], spatial context [11, 33, 30], temporal context [45], and even color [51] and sound [32].”
- Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “To define the frame/shot-level loss LS, we propose to build on any SSL loss designed for images, such as classifying exemplars [15], solving jigsaw puzzles of image patches [43], or rotation prediction [18].”
- A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, methodology): “Methods applied to the entire image (R OTATION , E XEMPLAR ) outperform patch-based methods (J IGSAW , R EL .”
- Learning Representations by Maximizing Mutual Information Across Views (2019, methodology): “E.g., Doersch et al. [2015] and Noroozi and Favaro [2016] learn representations by learning to predict/reconstruct spatial structure.”
- Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “Although many of these self-supervised objectives formulate prediction in terms of missing pixel-level details such as context or color [14, 69, 46, 38, 64], another class of meth-ods defines prediction in higher-level, learned representation spaces [25, 16, 47, 49, 51].”
- S4L: Self-Supervised Semi-Supervised Learning (2019, background): “Follow-up papers [24, 26] generalize this idea for predicting a permutation of multiple randomly sampled and permuted patches.”
- Self-Supervised MultiModal Versatile Networks (2020, background): “single modality), such as predicting the relative position of patches [18, 59], colorization [95], predicting orientation [27] or invariance to transformation [20, 36].”
- Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, background): “There are other approaches to self-supervised learning that are based on handcrafted pretext tasks [52, 31, 53, 54, 27, 55].”
- Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “In particular, relative patch prediction [23, 40], colorizing grayscale images [41, 42], image inpainting [43], image jigsaw puzzle [44], image super-resolution [45], and geometric transformations [46, 47] have been shown to be useful.”
- VirTex: Learning Visual Representations from Textual Annotations (2020, background): “Early works on self-supervised learning proposed hand-crafted pretext tasks, such as context prediction [15], colorization [17, 18], solving jigsaw puzzles [47], predicting rotation [19], inpainting [16], clustering [27], and generative modeling [48].”
- Contrastive Multiview Coding (2019, neutral): “mageNet-Labels 19.3 36.3 44.2 48.3 50.5 Random 11.6 17.1 16.9 16.3 14.1 Data-Init [31] 17.5 23.0 24.5 23.2 20.6 Context [14] 16.2 23.3 30.2 31.7 29.6 Colorization [62] 13.1 24.8 31.0 32.6 31.8 Jigsaw [43] 19.2 30.1 34.7 33.9 28.3 BiGAN [15] 17.7 24.5 31.0 29.9 28.0 SplitBrainy[63] 17.7 29.3 35.4 35.”
- Representation Learning with Contrastive Predictive Coding (2018, methodology): “The ImageNet dataset has been used to evaluate unsupervised vision models by many authors [28, 11, 35, 10, 29, 36].”
- BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “The method is similar to the masked version of Jigsaw pretraining (Noroozi and Favaro, 2016).”
- Unsupervised Feature Learning via Non-parametric Instance Discrimination (2018, methodology): “To learn a representation of images, the tasks could be: predicting the context [2], counting the objects [28], filling in missing parts of an image [31], recovering colors from grayscale images [47], or even solving a jigsaw puzzle [27].”
- Unsupervised Representation Learning by Predicting Image Rotations (2018, methodology): “ImageNet labels from (Bojanowski & Joulin, 2017) 59.7 59.7 Random from (Noroozi & Favaro, 2016) 27.1 12.0 Tracking Wang & Gupta (2015) 38.8 29.8 Context (Doersch et al., 2015) 45.6 30.4 Colorization (Zhang et al., 2016a) 40.7 35.2 Jigsaw Puzzles (Noroozi & Favaro, 2016) 45.3 34.6 BIGAN (Donahue et…”