Learning Representations for Automatic Colorization

By Gustav Larsson, Michael Maire, Gregory Shakhnarovich. 2016

1,061 citations recorded by the source corpus.

We develop a fully automatic image colorization system. Our approach leverages recent advances in deep networks, exploiting both low-level and semantic representations. As many scene elements naturally appear according to multimodal color distributions, we train our model to predict per-pixel color histograms. This intermediate output can be used to automatically generate a color image, or further manipulated prior to image formation. On both fully and partially automatic colorization tasks, we outperform existing methods. We also explore colorization as a vehicle for self-supervised visual representation learning.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

What makes ImageNet good for transfer learning? (2016, background): “ auxiliary tasks in an effort to make the CNNs “work harder”. Attempted selfsupervised tasks include predictions of ego-motion [1,16], spatial context [8,31,28], temporal context [41], and even color [45,23] and sound [30]. While features learned using these methods often come close to ImageNet perf”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “In particular, relative patch prediction [23, 40], colorizing grayscale images [41, 42], image inpainting [43], image jigsaw puzzle [44], image super-resolution [45], and geometric transformations [46, 47] have been shown to be useful.”
Unsupervised Representation Learning by Predicting Image Rotations (2018, background): “For example, in order to learn features, Zhang et al. (2016a) and Larsson et al. (2016) train ConvNets to colorize gray scale images, Doersch et al. (2015) and Noroozi & Favaro (2016) predict the relative position of image patches, and Agrawal et al. (2015) predict the egomotion (i.e., self-motion)…”
Learning Features by Watching Objects Move (2016, background): “Other pretext tasks include predicting color channels from luminance [25, 51] or vice versa [52], and predicting sounds from video frames [7,33].”
Colorful Image Colorization (2016, methodology): “c Labeled (MB) (MB) (ms) (%) (%) (%) Real (%) Ground Truth { { { 100 100 68.3 50 Gray { { { 89.1 58.0 52.7 { Random { { { 84.2 57.3 41.0 13.0 4.4 Dahl [2] { { { 90.4 58.9 48.7 18.3 2.8 Larsson et al. [23] 588 495 122.1 91.7 65.9 59.4 27.2 2.7 Ours (L2) 129 127 17.8 91.2 64.4 54.9 21.2 2.5 Ours (L2, ”
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (2018, methodology): “The algorithms are from pix2pix [20], Lars-son et al. [28], and variants from Zhang et al. [58].”
Image-to-Image Translation with Conditional Adversarial Networks (2016, methodology): “We trained colorization on ImageNet [48], and tested on the test split introduced by [58, 32].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, neutral): “Prior pretext tasks include image colorization [9, 30, 38, 39, 86, 87], orientation prediction [20], affine transform prediction [85], predicting contextual image patches [10], reordering image patches [5, 21, 53, 54, 56], counting visual primitives [55], or their combinations [11].”
Multi-task Self-Supervised Visual Learning (2017, background): “In the last few years, a considerable number of such tasks have been proposed [1, 2, 6, 7, 8, 17, 20, 21, 23, 25, 26, 27, 28, 29, 31, 39, 40, 42, 43, 46, 47], such as asking a neural network to colorize grayscale images, ﬁll in image holes, solve jigsaw puzzles made from image patches, or predict…”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “Many pretext tasks have been designed for images that exploit their spatial structure [12, 48–50], color information [11, 37, 38, 74], illumination [16], rotation [24] etc.”
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction (2016, methodology): “For example, networks trained on RGB images have been shown to perform well in tasks such as image segmentation [40], semantic segmentation [24], image colorization [42, 23, 16], depth prediction [7], surface normal prediction [7, 37], and image stylization [19].”
CrossTransformers: spatially-aware few-shot transfer (2020, background): “Our work on SimCLR episodes inherits from a line of self-supervised learning research, which typically deal with transfer from pretext tasks to semantic ones and must therefore represent more than their training data [3, 13, 15, 21, 24, 30, 36, 47, 60, 101, 102].”
Deep Clustering for Unsupervised Learning of Visual Features (2018, background): “Appart from spatial and temporal coherence, many other signals have been explored: image colorization [33, 71], cross-channel prediction [72], sound [44] or instance counting [43].”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “Many self-supervised methods manipulate the input data to extract a supervised signal in the form of a pretext task [1, 13, 26, 28, 30, 36, 38, 41, 42, 47, 48, 57].”
Representation Learning by Learning to Count (2017, methodology): “For example, some methods recover color from gray scale images and vice versa [43, 21, 44, 22], recover a whole patch from the surrounding pixels [33], or recover the relative location of patches [9, 29].”
Colorization as a Proxy Task for Visual Understanding (2017, methodology): “We consider both a regression loss for L*a*b color values [21, 43, 13], as well as a KL divergence loss for hue/chroma histograms [21].”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, background): “While only considering works that apply to a single, static image we can indicatively list successes of CNNs in superresolution [24], colorization [55], boundary detection [6,28,49,107], symmetry detection [91], interest point detection [109], image descriptors [35, 92, 110], surface Input…”
S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, background): “Different pretext tasks have been proposed like image rotations [17], image colourization [20], clustering [6], and solving jigsaw puzzles from image patch permutations [28].”