Scribbler: Controlling Deep Image Synthesis with Sketch and Color

By Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, James Hays. 2016

529 citations recorded by the source corpus.

Several recent works have used deep convolutional networks to generate realistic imagery. These methods sidestep the traditional computer graphics rendering pipeline and instead generate imagery at the pixel level by learning from large collections of photos (e.g. faces or bedrooms). However, these methods are of limited utility because it is difficult for a user to control what the network produces. In this paper, we propose a deep adversarial image synthesis architecture that is conditioned on sketched boundaries and sparse color strokes to generate realistic cars, bedrooms, or faces. We demonstrate a sketch based image synthesis system which allows users to scribble over the sketch to indicate preferred color for objects. Our network can then generate convincing images that satisfy both the color and the sketch constraints of user. The network is feed-forward which allows users to see the effect of their edits in real time. We compare to recent work on sketch to image synthesis and show that our approach generates more realistic, diverse, and controllable outputs. The architecture is also effective at user-guided colorization of grayscale images.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, background): “Scribbler [41] is a learning-based model where images are generated using line sketches as the input.”
Free-Form Image Inpainting With Gated Convolution (2018, background): “Scribbler network [Sangkloy et al. 2017] explored a deep adversarial image synthesis architecture conditioning on sketched boundaries and sparse color strokes to generate realistic cars, bedrooms, or faces.”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “Recent works [21, 45, 57] also provide user interfaces for creating novel imagery from low-level cues such as color and sketch.”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, methodology): “Similar ideas have been applied to various tasks such as generating photographs from sketches [40] or from attribute and semantic layouts [22].”
Stacked Generative Adversarial Networks (2016, background): “This approach, however, might introduce unnecessary complexity to the discriminator and increase model instability [46, 54].”
Toward Multimodal Image-to-Image Translation (2017, methodology): “However, the quality has been attained at the expense of multimodality, as the generator learns to largely ignore the random noise vector when conditioned on a relevant context [20, 34, 40, 45, 47, 55].”
CyCADA: Cycle-Consistent Adversarial Domain Adaptation (2017, background): “Recent work (Isola et al., 2016; Sangkloy et al., 2016; Karacan et al., 2016) adopt conditional GANs (Mirza & Osindero, 2014) for these image-to-image translation problems (Isola et al., 2016), but they require input-output image pairs for training, which is in general not available in domain…”
Seeing What a GAN Cannot Generate (2019, background): “Generative Adversarial Networks [15] have enabled many computer vision and graphics applications such as generation [7, 21, 22], image and video manipulation [19, 20, 30, 35, 39, 41, 52], object recognition [6, 42], and textto-image translation [33, 45, 49].”
Real-time user-guided image colorization with learned deep priors (2017, background): “…results for various image processing tasks, such as photo enhancement (Yan et al. 2016), sketch simplification (Simo-Serra et al. 2016), style transfer (Gatys et al. 2016; Selim et al. 2016), in-painting (Pathak et al. 2016), image blending (Zhu et al. 2015) and denoising (Gharbi et al. 2016).”
SinGAN: Learning a Generative Model From a Single Natural Image (2019, background): “Examples include interactive image editing [61, 10], sketch2image [8, 43], and other image-to-image translation tasks [62, 52, 54].”
DA-GAN: Instance-Level Image Translation by Deep Attention Generative Adversarial Networks (2018, methodology): “Similar ideas have been applied to various tasks such as generating photographs from sketches [29] or from attribute and semantic layouts [15].”
Generative Adversarial Network in Medical Imaging: A Review (2018, methodology): “The success of GANs in unsupervised anomaly detection (Schlegl et al., 2017) can help achieve the task of detecting abnormalities in medical images in an unsupervised manner.”
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, methodology): “They have also achieved strong performance on image editing using user sketch or color (Jo & Park, 2019; Liu et al., 2021; Sangkloy et al., 2017).”
AgileGAN: Stylizing Portraits by Inversion-Consistent Transfer Learning (2021, methodology): “Similar ideas have been applied to various tasks, such as sketches-to-photographs [Sangkloy et al. 2017] and attribute-to-image [Karacan et al.”
Swapping Autoencoder for Deep Image Manipulation (2020, methodology): “Many applications have been successfully built with this framework, including image inpainting [59, 30, 76, 51], photo colorization [82, 47, 84, 21], texture and geometry synthesis [85, 18, 74], sketch2photo [65], semantic image synthesis and editing [73, 62, 10, 58].”
Faceshop (2018, background): “Many of these global editing techniques build on image-to-image translation networks that learn to translate images from a source domain into a target domain [Chen and Koltun 2017; Isola et al. 2017; Sangkloy et al. 2017; Wang et al. 2017; Zhu et al. 2017].”
Learning to Sketch with Shortcut Cycle Consistency (2018, unknown): “We tried different state-of-the-art cross-domain translation models [13, 8, 33], but did not ﬁnd much difference between them.”