Deep exemplar-based colorization

By Mingming He, Dongdong Chen, Jing Liao, Pedro V. Sander, Lu Yuan. ACM Transactions on Graphics, 2018

336 citations recorded by the source corpus.

We propose the first deep learning approach for exemplar-based local colorization. Given a reference color image, our convolutional neural network directly maps a grayscale image to an output colorized image. Rather than using hand-crafted rules as in traditional exemplar-based methods, our end-to-end colorization network learns how to select, propagate, and predict colors from the large-scale data. The approach performs robustly and generalizes well even when using reference images that are unrelated to the input grayscale image. More importantly, as opposed to other learning-based colorization methods, our network allows the user to achieve customizable results by simply feeding different references. In order to further reduce manual effort in selecting the references, the system automatically recommends references with our proposed image retrieval algorithm, which considers both semantic and luminance information. The colorization can be performed fully automatically by simply picking the top reference suggestion. Our approach is validated through a user study and favorable quantitative comparisons to the-state-of-the-art methods. Furthermore, our approach can be naturally extended to video colorization. Our code and models are freely available for public use.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3149e4ec173374bae629188aa23b2020f4c6c8eb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Swapping Autoencoder for Deep Image Manipulation (2020, methodology): “Many applications have been successfully built with this framework, including image inpainting [59, 30, 76, 51], photo colorization [82, 47, 84, 21], texture and geometry synthesis [85, 18, 74], sketch2photo [65], semantic image synthesis and editing [73, 62, 10, 58].”
Neural Style Transfer: A Review (2017, background): “During the period of preparing this review, we are also delighted to find that related researches on NST also bring new inspirations for other areas [127], [128], [129], [130], [131] and accelerate the development of a wider vision community.”
Gated Context Aggregation Network for Image Dehazing and Deraining (2018, background): “In the future, we will try more facy losses used in [6, 19] and consider to extend to video dehazing like [5].”
SDEdit: Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “…an image based on a user input, and have demonstrated success on a variety of tasks including image synthesis and editing [42, 11, 12, 59, 39, 73], sketch2photo [47], inpainting [40, 25, 65, 35], photo colorization [67, 34, 68, 22], semantic image texture and geometry synthesis [69, 20, 61].”
Cross-Domain Correspondence Learning for Exemplar-Based Image Translation (2020, methodology): “Our work is inspired by the recent exemplar-based image colorization [48, 13], but we solve a more general problem: translating images between distinct domains.”
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “, 2018), photo colorization (Zhang et al., 2016; Larsson et al., 2016; Zhang et al., 2017; He et al., 2018), semantic image texture and geometry synthesis (Zhou et al.”