Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation

By Arna Ghosh, Richard Zhang, Puneet Kumar Dokania, Oliver Wang, Alexei A. Efros, Philip H. S. Torr, Eli Shechtman. 2019

151 citations recorded by the source corpus.

We propose an interactive GAN-based sketch-to-image translation method that helps novice users easily create images of simple objects. The user starts with a sparse sketch and a desired object category, and the network then recommends its plausible completion(s) and shows a corresponding synthesized image. This enables a feedback loop, where the user can edit the sketch based on the network's recommendations, while the network is able to better synthesize the image that the user might have in mind. In order to use a single model for a wide array of object classes, we introduce a gating-based approach for class conditioning, which allows us to generate distinct classes without feature mixing, from a single generator network.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GAN Inversion: A Survey (2021, background): “existing interactive image processing tasks such as sketchto-image generation [141], [142], [143] and sketch based image retrieval [144], [145], which usually require denselabeled datasets.”
UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis (2021, background): “With the increasing demand for flexible conditional image synthesis, various kinds of control signals have been introduced into this field, which can be divided into three main modalities: (i) textual controls (TC), including the class labels [1] and natural language descriptions [62, 54]; (ii) visu”
M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers (2021, background): “The image-to-image translation task generates photo-realism images from visual controls, such as a sketch map [17, 61], semantic label map [26, 4, 59], human pose [38] or another image for style transfer [15, 27].”
Design Guidelines for Prompt Engineering Text-to-Image Generative Models (2021, background): “In recent years, systems embedding generative models have been successfully applied to domains such as image generation, poetry, and music.[14, 15, 18, 19]”
Towards Open-World Text-Guided Face Image Generation and Manipulation (2021, background): “, sketch [1], semantic label [2], [3], or textual description [4], [5].”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, background): “Furthermore, attempts have been made to directly recognise partial sketches [27] and achieve sketch-to-photo generation [15] from incomplete sketch input, where both works involve a sketch-completion module based on image-to-image translation.”
TediGAN: Text-Guided Diverse Image Generation and Manipulation (2020, unknown): “To make image generation and manipulation more readily and user-friendly, recent studies have been focusing on the image synthesis conditioned on a variety of guidance, such as sketch [12, 51], semantic label [15, 49], or textual description [35, 52].”
Deep Learning for Free-Hand Sketch: A Survey (2020, background): “sign novel images in various practical applications: sketchbased photo editing [18], [220], [221], sketch to painting generation [222], cloth design [223], [224], sketch to natural photo generation [225], etc.”
Alleviating Semantics Distortion in Unsupervised Low-Level Image-to-Image Translation via Structure Consistency Constraint (2022, methodology): “It has been applied to various vision tasks [13, 46, 49, 59, 65].”
Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval (2020, unknown): “In addition to the four baselines, following the recent direction of dealing with partial sketches [12, 22], we tried a two stage framework for our early retrieval objective referred as TS in Table 1.”
Text as Neural Operator:Image Manipulation by Text Instruction (2020, background): “Examples of recent contributions are: text image residual gating (TIRG) [48], feature-wise linear modulation (FiLM) [42], and feature-wise gating [9].”
Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches (2020, methodology): “By introducing masks, they are extended to image completion tasks to modify the specified photo areas [36, 22, 12] or provide users with sketch recommendations [7].”