Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles

By Mehdi Noroozi, Paolo Favaro. 2016

3,276 citations recorded by the source corpus.

We propose a novel unsupervised learning approach to build features suitable for object detection and classification. The features are pre-trained on a large dataset without human annotation and later transferred via fine-tuning on a different, smaller and labeled dataset. The pre-training consists of solving jigsaw puzzles of natural images. To facilitate the transfer of features to other tasks, we introduce the context-free network (CFN), a siamese-ennead convolutional neural network. The features correspond to the columns of the CFN and they process image tiles independently (i.e., free of context). The later layers of the CFN then use the features to identify their geometric arrangement. Our experimental evaluations show that the learned features capture semantically relevant content. We pre-train the CFN on the training set of the ILSVRC2012 dataset and transfer the features on the combined training and validation set of Pascal VOC 2007 for object detection (via fast RCNN) and classification. These features outperform all current unsupervised features with \(51.8\,\%\) for detection and \(68.6\,\%\) for classification, and reduce the gap with supervised learning (\(56.5\,\%\) and \(78.2\,\%\) respectively).

Subfields: Computer Vision and Pattern Recognition, Geology, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Ming-Hsuan Yang
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Andrea Vedaldi

In Geology

  1. Jitendra Malik
  2. Ishan Misra
  3. Armand Joulin
  4. Leonidas J. Guibas
  5. Philipp Krahenbuhl
  6. Rohit Girdhar
  7. Andrea Vedaldi
  8. Kristen Grauman

In Representation & Self-Supervised Learning

  1. Ishan Misra
  2. Andrea Vedaldi
  3. Andrew Zisserman
  4. Trevor Darrell
  5. Dacheng Tao
  6. Armand Joulin
  7. Yann LeCun
  8. Kaiming He

Citations in the corpus