Semantic photo manipulation with a generative image prior

By David Bau, Hendrik Strobelt, William S. Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, Antonio Torralba. 2019

368 citations recorded by the source corpus.

Despite the recent success of GANs in synthesizing images conditioned on inputs such as a user sketch, text, or semantic labels, manipulating the high-level attributes of an existing natural photograph with GANs is challenging for two reasons. First, it is hard for GANs to precisely reproduce an input image. Second, after manipulation, the newly synthesized pixels often do not fit the original image. In this paper, we address these issues by adapting the image prior learned by GANs to image statistics of an individual image. Our method can accurately reconstruct the input image and synthesize new content, consistent with the appearance of the input image. We demonstrate our interactive system on several semantic image editing tasks, including synthesizing new objects consistent with background, removing unwanted objects, and changing the appearance of an object. Quantitative and qualitative comparisons against several existing methods demonstrate the effectiveness of our method.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims (2020, neutral): “, while limited in treatment, has primarily centered on comparing methods using human surveys [80].”
Paint by Word (2021, methodology): “One approach is to ﬁnd GAN la-tents to generate images that match a user’s painted intention: this can be done to match color [57] or a ﬁnite vocabulary of visual concepts [5, 4, 32] at a given location.”
Seeing What a GAN Cannot Generate (2019, background): “Prior work has found that inversions of GAN generators are useful for photo manipulation [2, 8, 31, 51] and unsupervised feature learning [10, 12].”
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation (2020, methodology): “[5, 3] utilized semantic segmentation to analyze Progressive GAN [16] and detect causal units that control the presence of certain objects through ablation.”
Image2StyleGAN++: How to Edit the Embedded Images? (2019, background): “Built on the understanding of how an object is “drawn” by the generator, they further designed a semantic image editing system that can add, remove or change the appearance of an object in an input image [3].”
ClipGen: A Deep Generative Model for Clipart Vectorization and Synthesis (2021, background): “The latent space can be explored through predefined interactions [42] or freeform interactions [43], [44] for synthesizing images that meet the users’ requirements.”
Imagic: Text-Based Real Image Editing with Diffusion Models (2022, methodology): “Multiple techniques for applying such manipulations on real images were suggested, including optimization-based methods [1, 2, 25], encoder-based methods [4, 48, 64], and methods adjusting the model per input [5, 9, 15, 49].”
TTT++: When Does Self-Supervised Test-Time Training Fail or Thrive? (2021, background): “Adapting machine learning models based on test samples has garnered growing interests in both generative problems such as super-resolution [13], image synthesis [14] and image manipulation [15], and discriminative problems like image classiﬁcation [16].”
InOut: Diverse Image Outpainting via GAN Inversion (2021, methodology): “In this work, we tackle the outpainting problem by inverting generative adversarial networks (GANs) [1,4,7,37].”
In-Domain GAN Inversion for Real Image Editing (2020, methodology): “To better apply well-trained GANs to real-world applications, GAN inversion enables real image editing from the latent space [31,23,2].”
Navigating the GAN Parameter Space for Semantic Image Editing (2020, methodology): “This paradigm has been successfully used for super-resolution [22], visual editing [35, 2], image restoration tasks [9, 23], e.”
Test-Time Training for Out-of-Distribution Generalization (2019, background): “Concurrent work Bau et al. (2019) improves photo manipulation with a generative adversarial network, by adapting its image prior to the statistics of the input image.”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, neutral): “TTA method can also vary the pre-trained generative model for style transfer and data generation [261, 262, 379].”
Transforming and Projecting Images into Class-conditional Generative Networks (2020, methodology): “Many prior works have investigated using trained generative models as a tool to edit images [60, 9, 5, 6].”
Test-Time Training with Self-Supervision for Generalization under Distribution Shifts (2019, background): “More recently, Bau et al. (2019) improve photo manipulation by adapting a pre-trained GAN to the statistics of the input image.”
EditGAN: High-Precision Semantic Image Editing (2021, methodology): “Previous papers studied encoder-based methods [58, 59, 60, 61, 62], used primarily optimization-based techniques [63, 64, 65, 66, 67, 68, 69, 26], and developed hybrid approaches [63, 24, 25, 70, 71].”
Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (2024, methodology): “Various approaches employ generative priors, including GAN inversion [2, 4, 27, 55, 60], GAN encoders [9, 100], or using GAN as the core module for IR [77, 84].”