Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches

By Shuai Yang, Zhangyang Wang, Jiaying Liu, Zongming Guo. 2020

78 citations recorded by the source corpus.

Sketch-based image editing aims to synthesize and modify photos based on the structural information provided by the human-drawn sketches. Since sketches are difficult to collect, previous methods mainly use edge maps instead of sketches to train models (referred to as edge-based models). However, sketches display great structural discrepancy with edge maps, thus failing edge-based models. Moreover, sketches often demonstrate huge variety among different users, demanding even higher generalizability and robustness for the editing model to work. In this paper, we propose Deep Plastic Surgery, a novel, robust and controllable image editing framework that allows users to interactively edit images using hand-drawn sketch inputs. We present a sketch refinement strategy, as inspired by the coarse-to-fine drawing process of the artists, which we show can help our model well adapt to casual and varied sketches without the need for real sketch training data. Our model further provides a refinement level control parameter that enables users to flexibly define how "reliable" the input sketch should be considered for the final output, balancing between sketch faithfulness and output verisimilitude (as the two goals might contradict if the input sketch is drawn poorly). To achieve the multi-level refinement, we introduce a style-based module for level conditioning, which allows adaptive feature representations for different levels in a singe network. Extensive experimental results demonstrate the superiority of our approach in improving the visual quality and user controllablity of image editing over the state-of-the-art methods.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, methodology): “Some artistic application of sketch includes image editing [66], animation auto-completion [63], etc. Sketches have lately been used to create Pictionary-style competitive drawing games [4].”
EnlightenGAN: Deep Light Enhancement Without Paired Supervision (2019, methodology): “When applying GANs to image restoration and enhancement, most existing works use paired training data as well, such as super resolution [28], artistic style transfer and image editing [29], [30], deraining [31] and dehazing [32].”
Deep Learning for Free-Hand Sketch: A Survey (2020, background): “images in various practical applications: sketch-based photo editing [22], [263], [264], sketch to painting generation [276],”