EditGAN: High-Precision Semantic Image Editing

By Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, Sanja Fidler. 2021

242 citations recorded by the source corpus.

Generative adversarial networks (GANs) have recently found applications in image editing. However, most GAN based image editing methods often require large scale datasets with semantic segmentation annotations for training, only provide high level control, or merely interpolate between different images. Here, we propose EditGAN, a novel method for high quality, high precision semantic image editing, allowing users to edit images by modifying their highly detailed part segmentation masks, e.g., drawing a new mask for the headlight of a car. EditGAN builds on a GAN framework that jointly models images and their semantic segmentations, requiring only a handful of labeled examples, making it a scalable tool for editing. Specifically, we embed an image into the GAN latent space and perform conditional latent code optimization according to the segmentation edit, which effectively also modifies the image. To amortize optimization, we find editing vectors in latent space that realize the edits. The framework allows us to learn an arbitrary number of editing vectors, which can then be directly applied on other images at interactive rates. We experimentally show that EditGAN can manipulate images with an unprecedented level of detail and freedom, while preserving full image quality.We can also easily combine multiple edits and perform plausible edits beyond EditGAN training data. We demonstrate EditGAN on a wide variety of image types and quantitatively outperform several previous editing methods on standard editing benchmark tasks.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, background): “2019, 2020b], fine-grained semantic control [Härkönen et al. 2020; Ling et al. 2021; Wu et al. 2021], and recently alias-free generation enabling realistic animation [Karras et al.”
Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold (2023, methodology): “Most previous approaches gain controllability of GANs via prior 3D models [Deng et al. 2020; Ghosh et al. 2020; Tewari et al. 2020] or supervised learning that relies on manually annotated data [Abdal et al. 2021; Isola et al. 2017; Ling et al. 2021; Park et al. 2019; Shen et al. 2020].”
Third Time's the Charm? Image and Video Editing with StyleGAN3 (2022, background): “Specifically, StyleGAN generators [33–36] synthesize exceed-ingly realistic images, and enable editing [4,5,9,15,42,48, 83], and image-to-image translation [46,47,55], particularly in well-structured domains.”
Diffusion Models already have a Semantic Latent Space (2022, background): “Meanwhile, generative adversarial network Goodfellow et al. (2020) address their latent space for image editing (Ling et al. (2021), H¨ark¨onen et al. (2020), Chefer et al. (2021), Shen et al. (2020), Y¨uksel et al. (2021), Patashnik et al. (2021), Gal et al. (2021), Dai et al. (2019), Xu et al.…”
MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing (2023, background): “As a result, text-guided image editing [21, 3, 8, 16, 14] has recently gained more popularity compared to other mask-based image editing techniques [20, 35, 23].”
High-fidelity GAN Inversion with Padding Space (2022, background): “Existing studies on GAN interpretation have affirmed that, pre-trained GAN models own great potential in a range of downstream applications, such as object classification [11, 47], semantic segmentation [51], video generation [24], and image editing [5, 13,15,21,29,32,39,40,44,48,52,53].”
Paint by Example: Exemplar-based Image Editing with Diffusion Models (2022, background): “Another popular direction of works [5,18,36,65,73,75] utilize semantic mask to control the editing.”
Improving Alignment and Robustness with Circuit Breakers (2024, unknown): “This includes research ranging from uncovering emergent interpretable structures in intermediate representations [77, 46, 10], to the identification and modification of embedded knowledge [48, 42, 43], as well as steering model outputs [66, 7, 33, 24, 65].”