Colorization as a Proxy Task for Visual Understanding

By Gustav Larsson, Michael Maire, Gregory Shakhnarovich. 2017

520 citations recorded by the source corpus.

We investigate and improve self-supervision as a drop-in replacement for ImageNet pretraining, focusing on automatic colorization as the proxy task. Self-supervised training has been shown to be more promising for utilizing unlabeled data than other, traditional unsupervised learning methods. We build on this success and evaluate the ability of our self-supervised network in several contexts. On VOC segmentation and classification tasks, we present results that are state-of-the-art among methods not using ImageNet labels for pretraining representations. Moreover, we present the first in-depth analysis of self-supervision via colorization, concluding that formulation of the loss, training details and network architecture play important roles in its effectiveness. This investigation is further expanded by revisiting the ImageNet pretraining paradigm, asking questions such as: How much training data is needed? How many labels are needed? How much do features change when fine-tuned? We relate these questions back to self-supervision by showing that colorization provides a similarly powerful supervisory signal as various flavors of ImageNet pretraining.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, background): “These proxy tasks leverage structure within the data to generate supervised tasks automatically ( e.g . colorizing images [20] or reconstructing masked words in text [12]).”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “Other works showed that prediction of color (Zhang et al., 2016; Larsson et al., 2017) and image orientation (Gidaris et al.”
Learning Visual Representations with Caption Annotations (2020, background): “tasks consist in predicting missing pieces in manipulated images, for instance context prediction [15], colorization [13,36,66], inpainting of missing portions [46], prediction of image rotations [19], spotting artifacts [31], or cross-channel prediction [67].”
Unsupervised Feature Learning via Non-parametric Instance Discrimination (2018, background): “Note that the results for colorization-based pretraining are from a deeper ResNet-152 network [19].”
Unsupervised Representation Learning by Predicting Image Rotations (2018, background): “…on other vision tasks, such as object recognition, object detection, and semantic segmentation (Zhang et al., 2016a; Larsson et al., 2016; Zhang et al., 2016b; Larsson et al., 2017; Doersch et al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2017; Pathak et al., 2016a; Doersch & Zisserman, 2017).”
Self-Supervised Learning of Pretext-Invariant Representations (2019, neutral): “Prior pretext tasks include image colorization [9, 30, 38, 39, 86, 87], orientation prediction [20], affine transform prediction [85], predicting contextual image patches [10], reordering image patches [5, 21, 53, 54, 56], counting visual primitives [55], or their combinations [11].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, neutral): “…al., 2018; Mundhenk et al., 2017), clustering (Caron et al., 2018), noise-as-targets (Bojanowski & Joulin, 2017), colorization (Zhang et al., 2016; Larsson et al., 2017), generation (Jenni & Favaro, 2018; Ren & Lee, 2018; Donahue et al., 2017), geometry (Dosovitskiy et al., 2016; Gidaris et al.,…”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “Many pretext tasks have been designed for images that exploit their spatial structure [12, 48–50], color information [11, 37, 38, 74], illumination [16], rotation [24] etc.”
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction (2016, background): “Autoencoder [19] reconstruction no no Denoising autoencoder [43] reconstruction suffers no Context Encoder [34] prediction no suffers Cross-Channel Encoder [47, 27] prediction no suffers Split-Brain Autoencoder prediction no no”
An Unsupervised Autoregressive Model for Speech Representation Learning (2019, background): “The autoregressive loss belongs to a large family of selfsupervised loss functions [8, 9, 10].”
Representation Learning by Learning to Count (2017, methodology): “For example, some methods recover color from gray scale images and vice versa [43, 21, 44, 22], recover a whole patch from the surrounding pixels [33], or recover the relative location of patches [9, 29].”
Self-labelling via simultaneous clustering and representation learning (2019, neutral): “, 2019), noise-as-targets (Bojanowski & Joulin, 2017), colorization (Zhang et al., 2016; Larsson et al., 2017), generation (Jenni & Favaro, 2018; Ren & Lee, 2018; Donahue et al.”
Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning (2018, background): “Related to an image colorization task (Larsson et al., 2017), x could be the gray-scale image and u the hue.”
Transitive Invariance for Self-Supervised Visual Representation Learning (2017, background): “k et al. [44] train a network to segment the foreground objects where are acquired via motion cues. On the other hand, common characteristics of different object instances can also be mined from data [9,63,64,30,31]. For example, relative positions of image patches [9] may reﬂect feasible spatial la”
Tracking Emerges by Colorizing Videos (2018, background): “Besides the core problem of colorizing images, colorization has been shown to be a useful side task to learn representations for images without supervision [9, 10].”
Improvements to Context Based Self-Supervised Learning (2017, methodology): “Since then, methods have been introduced that use context arrangement of image patches [9, 31], image completion [36], image colorization [48, 47, 24], motion segmentation [35], motion frame ordering [43, 44, 25], object counting [33] and a multi-task ensemble of many models [10, 21].”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, methodology): “Different types of image transformations are designed as supervision information to train 2DCNNs for image representation learning including image colorization [25], image rotation [12], and image denoise [2].”
Self-Supervised Feature Learning by Learning to Spot Artifacts (2018, background): “[24] demonstrate good transfer performance with models trained on the task of image colorization, i.”