Real-time user-guided image colorization with learned deep priors

By Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, Alexei A. Efros. 2017

633 citations recorded by the source corpus.

We propose a deep learning approach for user-guided image colorization. The system directly maps a grayscale image, along with sparse, local user "hints" to an output colorization with a Convolutional Neural Network (CNN). Rather than using hand-defined rules, the network propagates user edits by fusing low-level cues along with high-level semantic information, learned from large-scale data. We train on a million images, with simulated user inputs. To guide the user towards efficient input selection, the system recommends likely colors based on the input image and current user inputs. The colorization is performed in a single feed-forward pass, enabling real-time use. Even with randomly simulated user inputs, we show that the proposed system helps novice users quickly create realistic colorizations, and offers large improvements in colorization quality with just a minute of use. In addition, we demonstrate that the framework can incorporate other user "hints" to the desired colorization, showing an application to color histogram transfer.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Free-Form Image Inpainting With Gated Convolution (2018, background): “[50] propose colorization networks which can take user guidance as additional inputs.”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “Recent works [21, 45, 57] also provide user interfaces for creating novel imagery from low-level cues such as color and sketch.”
Rewriting a Deep Generative Model (2020, background): “Recently, thanks to rapid advances of deep generative models [26,45,31], learning-based image synthesis and editing methods have become widely-used tools in the community, enabling applications such as manipulating the semantics of an input scene [60,6,72,16], image colorization [83,34,52,85], photo”
The Perception-Distortion Tradeoff (2017, background): “-fake user studies, which examine the ability of human observers to tell whether x̂ is real or the output of an algorithm [48, 14, 6, 13, 49, 22, 10] (similarly to the idea underlying generative adversarial nets [9]).”
Toward Multimodal Image-to-Image Translation (2017, methodology): “For example, color and shape scribbles and other interfaces were used as conditioning in iGAN [54], pix2pix [20], Scribbler [40] and interactive colorization [51].”
Guide Me: Interacting with Deep Networks (2018, methodology): “Additionally, a method for sparse, user-guided colorization of grayscale images is proposed in [63].”
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “, 2018), photo colorization (Zhang et al., 2016; Larsson et al., 2016; Zhang et al., 2017; He et al., 2018), semantic image texture and geometry synthesis (Zhou et al.”
Deep Depth Completion of a Single RGB-D Image (2018, methodology): “To resolve the ambiguity between “missing” and “0 meter”, a binary mask indicating the pixels that have depth from sensor is added as an additional channel as suggested in Zhang et.al [79].”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “Colorization : The concept of colorization was initially introduced in [38], and subsequent studies [39], [40], [41] demonstrated its effectiveness as a pretext task for SSL. Color prediction offers the advantageous feature of requiring freely available training data.”
Generative Adversarial Networks–Enabled Human–Artificial Intelligence Collaborative Applications for Creative and Design Industries: A Systematic Review of Current Approaches and Trends (2021, background): “Architecturally, the work presented by Zhang et al. (2017) is quite similar.”
Big but Imperceptible Adversarial Perturbations via Semantic Manipulation (2019, background): “[34] demonstrated an improved and diverse image colorization by using input hints from users guided colorization process.”
U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation (2019, background): “…computer vision because of its wide range of applications including image inpainting (Pathak et al. (2014); Iizuka et al. (2017)), super resolution (Dong et al. (2016); Kim et al. (2016)), colorization (Zhang et al. (2016; 2017)) and style transfer (Gatys et al. (2016); Huang & Belongie (2017)).”
Editing Conditional Radiance Fields (2021, background): “Recent works have focused on integrating user interactions into deep networks either through optimization [1, 9, 81, 10] or a feed-forward network with user-guided inputs [79, 57, 52, 49].”
Image-to-Image Translation: Methods and Applications (2021, neutral): “image colorization [1], [87], [21], [23], [24], [25] \ [140], [22], [26] \ [103] \ \ \ \ \ \”
Colorization Transformer (2021, background): “…(Levin et al., 2004; Ironi et al., 2005; Huang et al., 2005; Yatziv & Sapiro, 2006; Qu et al., 2006; Luan et al., 2007; Tsaftaris et al., 2014; Zhang et al., 2017; Ci et al., 2018) and exemplar-based techniques that involve identifying a reference source image to copy colors from (Reinhard et…”
Knowledge As Priors: Cross-Modal Knowledge Generalization for Datasets Without Superior Knowledge (2020, methodology): “…the original images collected with the green screen background ( G ) in FreiHAND, together with their post-processed results using three different strategies: harmonization [46] ( H ), colorization auto [57] ( A ), colorization sample [57] ( S ), as three different domains contained by FreiHAND.”
TextureGAN: Controlling Deep Image Synthesis with Texture Patches (2017, background): “…perceptually controllable interfaces, ranging from high-level attributes, such as object classes [28], object poses [4], natural language descriptions [30], to fine-grained details, such as segmentation masks [16], sketches [12, 31], color scribbles [31, 41], and cross-domain images [9, 35].”
Using latent space regression to analyze and leverage compositionality in GANs (2021, result): “…resolution, and sample a threshold t from the uniform distribution in range [0.3, 1.0] to create the binary mask: m = 1 [Upsample(u) > t] xm = m⊗ x (11) We also experimented with masks comprised of random rectangular patches (Zhang et al., 2017), but obtained qualitatively similar results.”
FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images (2019, methodology): “Both the automatic and sampling variant of [31] were used.”