Adversarial Scene Editing: Automatic Object Removal from Weak Supervision

By Rakshith Shetty, Mario Fritz, Bernt Schiele. 2018

101 citations recorded by the source corpus.

While great progress has been made recently in automatic image manipulation, it has been limited to object centric images like faces or structured scene datasets. In this work, we take a step towards general scene-level image editing by developing an automatic interaction-free object removal model. Our model learns to find and remove objects from general scene images using image-level labels and unpaired data in a generative adversarial network (GAN) framework. We achieve this with two key contributions: a two-stage editor architecture consisting of a mask generator and image in-painter that co-operate to remove objects, and a novel GAN based prior for the mask generator that allows us to flexibly incorporate knowledge about object shapes. We experimentally show on two datasets that our method effectively removes a wide variety of objects using weak supervision only.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

MONet: Unsupervised Scene Decomposition and Representation (2019, background): “…learning from the structure and statistics of the data, these methods often rely on heuristics (such as region masks, neighboring patches, or reference frames for colorization), and occasionally on explicit supervision (e.g. to recognize the presence of a type of object in Shetty et al. [2018]).”
Not Using the Car to See the Sidewalk — Quantifying and Controlling the Effects of Context in Classification and Segmentation (2018, methodology): “We base our in-painting network on the model proposed in [21], since this inpainter is directly optimized for removal, and can better handle irregular masks used in removal [21].”
Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, methodology): “loss (details in the supplementary material). Generating counterfactual examples. Counterfactual examples can be generated using existing annotation in COCO. Agarwal et al. [1] used the inpainter GAN [59] to edit images by masking selected objects. This only requires the original labels and bounding”
Noise or Signal: The Role of Image Backgrounds in Object Recognition (2020, methodology): “Other works solve this problem, called image inpainting, either using exemplar-based methods [CPT04] or using deep learning [Yu+18; SFS18].”
StoryGAN: A Sequential Conditional GAN for Story Visualization (2018, background): “Another related task is textual image editing, which edits an input image according to a textual editing query [3, 30, 4, 9].”
Recurrent Feature Reasoning for Image Inpainting (2020, background): “These algorithms have a wide range of applications in photo editing, de-captioning and other scenarios where people might want to remove unwanted objects from their photos [23, 22, 1, 4].”
Media Forensics and DeepFakes: An Overview (2020, background): “Besides conventional manipulations, like composition [18], copy-move [19], object removal [20], inpainting [21], a large number of new tasks can be performed.”