Colorful Image Colorization

By Richard Zhang, Phillip Isola, Alexei A. Efros. 2016

3,751 citations recorded by the source corpus.

Given a grayscale photograph as input, this paper attacks the problem of hallucinating a plausible color version of the photograph. This problem is clearly underconstrained, so previous approaches have either relied on significant user interaction or resulted in desaturated colorizations. We propose a fully automatic approach that produces vibrant and realistic colorizations. We embrace the underlying uncertainty of the problem by posing it as a classification task and use class-rebalancing at training time to increase the diversity of colors in the result. The system is implemented as a feed-forward pass in a CNN at test time and is trained on over a million color images. We evaluate our algorithm using a “colorization Turing test,” asking human participants to choose between a generated and ground truth color image. Our method successfully fools humans on 32 % of the trials, significantly higher than previous methods. Moreover, we show that colorization can be a powerful pretext task for self-supervised feature learning, acting as a cross-channel encoder. This approach results in state-of-the-art performance on several feature learning benchmarks.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Masked Autoencoders Are Scalable Vision Learners (2021, background): “, masking pixels [49, 39, 6] or removing color channels [59].”
A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “, 2015), solving jigsaw puzzles (Noroozi & Favaro, 2016), colorization (Zhang et al., 2016) and rotation prediction (Gidaris et al.”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, background): “, eight positions [13], color bins [64]) by cross-entropy or margin-based losses.”
UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “…learning utilizes original data as its own source of supervision, which has been applied to many Computer Vision tasks, such as image colorization (Zhang et al., 2016), solving jigsaw puzzles (Noroozi & Favaro, 2016; Trinh et al., 2019), inpainting (Pathak et al., 2016), rotation prediction…”
What makes ImageNet good for transfer learning? (2016, background): “Attempted self-supervised tasks include predictions of ego-motion [1, 20], spatial context [11, 33, 30], temporal context [45], and even color [51] and sound [32].”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “Prior work has explored diverse self-supervision cues such as predicting the spatialcontext [12], colorization [71], equivariance to transformations [18, 44]; alongside unsupervised techniques such as clustering [6, 72], generative modelling [14, 32], and exemplar learning [15].”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, methodology): “In linear/MLP evaluation, widely used in for self-supervised representations (Doersch et al., 2015; Zhang et al., 2016; Noroozi & Favaro, 2016; Doersch & Zisserman, 2017), the weights of a pre-trained network are frozen, and a linear layer/MLP is trained on top to predict the labels.”
Learning Representations by Maximizing Mutual Information Across Views (2019, background): “Zhang et al. [2016] introduce the task of predicting color information that has been removed by converting images to grayscale.”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “Other works showed that prediction of color (Zhang et al., 2016; Larsson et al., 2017) and image orientation (Gidaris et al.”
S4L: Self-Supervised Semi-Supervised Learning (2019, background): “Among those, in [41] the authors propose to use grayscale image colorization as a pretext task.”
Learning Visual Representations with Caption Annotations (2020, background): “Often, proxy tasks consist in predicting missing pieces in manipulated images, for instance context prediction [15], colorization [13,36,66], inpainting of missing portions [46], prediction of image rotations [19], spotting artifacts [31], or cross-channel prediction [67].”
Self-Supervised MultiModal Versatile Networks (2020, background): “single modality), such as predicting the relative position of patches [18, 59], colorization [95], predicting orientation [27] or invariance to transformation [20, 36].”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, background): “There are other approaches to self-supervised learning that are based on handcrafted pretext tasks [52, 31, 53, 54, 27, 55].”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “Linear evaluation on ImageNet We first evaluate BYOL’s representation by training a linear classifier on top of the frozen representation, following the procedure described in [48, 74, 41, 10, 8], and appendix D.”
VirTex: Learning Visual Representations from Textual Annotations (2020, background): “Early works on self-supervised learning proposed hand-crafted pretext tasks, such as context prediction [15], colorization [17, 18], solving jigsaw puzzles [47], predicting rotation [19], inpainting [16], clustering [27], and generative modeling [48].”
Contrastive Multiview Coding (2019, methodology): “ethod conv1 conv2 conv3 conv4 conv5 ImageNet-Labels 19.3 36.3 44.2 48.3 50.5 Random 11.6 17.1 16.9 16.3 14.1 Data-Init [31] 17.5 23.0 24.5 23.2 20.6 Context [14] 16.2 23.3 30.2 31.7 29.6 Colorization [62] 13.1 24.8 31.0 32.6 31.8 Jigsaw [43] 19.2 30.1 34.7 33.9 28.3 BiGAN [15] 17.7 24.5 31.0 29.9 28”
Representation Learning with Contrastive Predictive Coding (2018, methodology): “Work shown useful [10, 11].”
BEiT: BERT Pre-Training of Image Transformers (2021, background): “Pioneering works design clever pretext tasks, such as predicting the patch orderings (Noroozi and Favaro, 2016), colorization (Zhang et al., 2016), and predicting rotation angles (Komodakis and Gidaris, 2018).”
Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “Standard protocols for selfsupervised learning are to either learn a linear classifier on frozen features [67, 26] or to finetune the features on downstream tasks.”