SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis

By Wengling Chen, James Hays. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

335 citations recorded by the source corpus.

Synthesizing realistic images from human drawn sketches is a challenging problem in computer graphics and vision. Existing approaches either need exact edge maps, or rely on retrieval of existing photographs. In this work, we propose a novel Generative Adversarial Network (GAN) approach that synthesizes plausible images from 50 categories including motorcycles, horses and couches. We demonstrate a data augmentation technique for sketches which is fully automatic, and we show that the augmented data is helpful to our task. We introduce a new network building block suitable for both the generator and discriminator which improves the information flow by injecting the input image at multiple scales. Compared to state-of-the-art image translation methods, our approach generates more realistic images and achieves significantly higher Inception Scores.

View Paper: https://www.semanticscholar.org/paper/bbe3d39adcb41ad2824204c0b0d299d77c2d8363

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

SinGAN: Learning a Generative Model From a Single Natural Image (2019, methodology): “Examples include interactive image editing [61, 10], sketch2image [8, 43], and other image-to-image translation tasks [62, 52, 54].”
Image Synthesis with a Single (Robust) Classifier (2019, methodology): “For example, GANs have been used to transform simple sketches [CH18; Par+19] into realistic images.”
Cali-Sketch: Stroke Calibration and Completion for High-Quality Face Image Generation from Poorly-Drawn Sketches (2019, methodology): “SketchyGAN [14] synthesizes plausible images of objects from 50 classes.”
Computer Vision with a Single (Robust) Classifier (2019, methodology): “For example, GANs have been used to transform simple sketches [CH18; Par+19] into realistic images.”
Deep Sketch-Guided Cartoon Video Inbetweening (2021, background): “Recently, with the emergence of deep learning, sketches can be directly mapped to realistic photographs by learning from data [15]–[17].”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, background): “That is, instead of costly operation like sketch deformation [58] or unfaithful approximation like edge/contour-map as soft ground-truths [9], users can effortlessly generate n most representative subsets to augment training for many downstream tasks.”
Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches (2020, background): “Some human-drawn sketch datasets have also been collected [24, 37] to train sketch-based models [3, 17].”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, background): “Sketch synthesis: T-GAN [102], Autopainter [104], Sketchy-GAN [105], CA-GAN [106]”
Deep Learning for Free-Hand Sketch: A Survey (2020, background): “Sketch generation can be solved in various frameworks, e . g ., variational autoencoder (VAE) [16], [59], [97], Generative Adversarial Network (GAN) [98], VAE-GAN [98], reinforcement learning (RL) [98], [99].”