Learning Image Representations by Completing Damaged Jigsaw Puzzles

By Dahun Kim, Donggeun Yoo, In-So Kweon. 2018

170 citations recorded by the source corpus.

In this paper, we explore methods of complicating selfsupervised tasks for representation learning. That is, we do severe damage to data and encourage a network to recover them. First, we complicate each of three powerful self-supervised task candidates: jigsaw puzzle, inpainting, and colorization. In addition, we introduce a novel complicated self-supervised task called "Completing damaged jigsaw puzzles" which is puzzles with one piece missing and the other pieces without color. We train a convolutional neural network not only to solve the puzzles, but also generate the missing content and colorize the puzzles. The recovery of the aforementioned damage pushes the network to obtain robust and general-purpose representations. We demonstrate that complicating the self-supervised tasks improves their original versions and that our final task learns more robust and transferable representations compared to the previous methods, as well as the simple combination of our candidate tasks. Our approach achieves state-of-the-art performance in transfer learning on PASCAL classification and semantic segmentation.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Revisiting Self-Supervised Visual Representation Learning (2019, background): “extend the “jigsaw puzzle” task by combining it with colorization and inpainting in [22].”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “Many self-supervised methods manipulate the input data to extract a supervised signal in the form of a pretext task [1, 13, 26, 28, 30, 36, 38, 41, 42, 47, 48, 57].”
Self-Supervised Learning: Generative or Contrastive (2020, methodology): “CDJP [61] CV C Jigsaw + Inpainting + Colorization End-to-end”
Improvements to Context Based Self-Supervised Learning (2017, methodology): “Since then, methods have been introduced that use context arrangement of image patches [9, 31], image completion [36], image colorization [48, 47, 24], motion segmentation [35], motion frame ordering [43, 44, 25], object counting [33] and a multi-task ensemble of many models [10, 21].”
Geographical Knowledge-Driven Representation Learning for Remote Sensing Images (2021, background): “For example, occlude an area in the image randomly and repair it with networks, from which a good image representation is learned [17, 18, 35, 40].”
Leveraging Large-Scale Uncurated Data for Unsupervised Pre-training of Visual Features (2019, background): “Many pretext tasks have been proposed [22, 31, 47, 51], exploiting, amongst others, spatial context [12, 24, 33, 34, 36], cross-channel prediction [27, 28, 55, 56], or the temporal structure of videos [1, 35, 46].”
Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles (2018, methodology): “2017) and multi-task of many models (Kim et al. 2018) have been proposed.”
Evaluation of Split-Brain Autoencoders for High-Resolution Remote Sensing Scene Classification (2018, background): “Some of the examples of these tasks are colorization [1], image inpainting [2], learning rotations [3], solving puzzles [4], [5], etc.”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, methodology): “Method Category Code Contribution GAN [83] Generation 3 Forerunner of GAN DCGAN [120] Generation 3 Deep convolutional GAN for image generation WGAN [121] Generation 3 Proposed WGAN which makes the training of GAN more stable BiGAN [122] Generation 3 Bidirectional GAN to project data into latent spac”
MixCo: Mix-up Contrastive Learning for Visual Representation (2020, methodology): “Contrastive Learning Although earlier self-supervised methods handcrafted pretext tasks [1, 10, 17, 20, 22, 33], most recent approaches that show remarkable performance are based on instance-level classification with contrastive loss [5, 6, 7, 12, 16, 26, 31].”