In-Domain GAN Inversion for Real Image Editing

By Jiapeng Zhu, Yujun Shen, Deli Zhao, Bolei Zhou. 2020

694 citations recorded by the source corpus.

Recent work has shown that a variety of semantics emerge in the latent space of Generative Adversarial Networks (GANs) when being trained to synthesize images. However, it is difficult to use these learned semantics for real image editing. A common practice of feeding a real image to a trained GAN generator is to invert it back to a latent code. However, existing inversion methods typically focus on reconstructing the target image by pixel values yet fail to land the inverted code in the semantic domain of the original latent space. As a result, the reconstructed image cannot well support semantic editing through varying the inverted code. To solve this problem, we propose an in-domain GAN inversion approach, which not only faithfully reconstructs the input image but also ensures the inverted code to be semantically meaningful for editing. We first learn a novel domain-guided encoder to project a given image to the native latent space of GANs. We then propose domain-regularized optimization by involving the encoder as a regularizer to fine-tune the code produced by the encoder and better recover the target image. Extensive experiments suggest that our inversion method achieves satisfying real image reconstruction and more importantly facilitates various image editing tasks, significantly outperforming start-of-the-arts.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “013 seconds to obtain the latent code in D2C, while the same takes 8 seconds [106] for StyleGAN2 (615× slower).”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, background): “However, as highlighted by [Tov et al. 2021; Zhu et al. 2020], this extended definition achieves higher reconstruction quality in exchange for lower editability.”
StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows (2020, methodology): “Finally, there is the idea to combine the two techniques and rst use the encoder network to obtain an approximate embedding and then re ne it with an optimization algorithm (Zhu et al. 2020, 2016).”
Designing an encoder for StyleGAN image manipulation (2021, methodology): “This task has been commonly referred to as GAN Inversion [31, 42, 45].”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, methodology): “To accelerate this optimization process, some methods [4, 42] have trained an encoder to infer an approximate vector in W+ which serves as a good initial point from which additional optimization is required.”
Paint by Word (2021, background): “Because several powerful editing methods are enabled by GAN latent manipulation, the problem of inverting a GAN is of widespread interest and is the subject of an active line of ongoing research [53, 26, 6, 1, 2, 14, 36, 52, 55, 16].”
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation (2020, methodology): “This may be done via latent optimization [1, 2] or by training an encoder [39, 38] based on reconstruction loss (LPIPS [37] or L2).”
Closed-Form Factorization of Latent Semantics in GANs (2020, methodology): “rsion Age Eyeglasses Pose Smile Gender Figure 6: Real image manipulation with respect to various facial attributes. All semantics are found with the proposed closed-form approach, SeFa. GAN inversion [27] is used to project the target real image back to the latent space of StyleGAN [16]. Table 3: Us”
HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing (2021, methodology): “Recent works [2, 61, 73, 75] have demonstrated the ex-”
Improved Transformer for High-Resolution GANs (2021, methodology): “In addition to the reconstruction loss and adversarial loss, our HiT-based VQ-VAE variant, namely VQ-HiT, is also trained with the perceptual loss [24] following the setup of [14, 78].”
Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis (2021, methodology): “Despite the various back-tracking methods (Zhu et al., 2016; Lipton & Tripathi, 2017; Zhu et al., 2020; Abdal et al., 2019), a well generalized G is arguably as important for the good inversions.”
AgileGAN: Stylizing Portraits by Inversion-Consistent Transfer Learning (2021, methodology): “We compared our hierarchical variational encoder to alternative embedding methods, including PSP encoder [Richardson et al. 2020], in-domain encoder [Zhu et al. 2020] and StyleGAN2 optimization [Karras et al. 2020b], by evaluating with these substitutes.”
Learning on Gradients: Generalized Artifacts Representation for GAN-Generated Images Detection (2023, neutral): “VGG16 [39], InceptionV3 [41], Resnet50 [15], DeeplabV3 [5], CLIP [35], ViT [10], discriminator of ProGAN [21], StyleGAN [22], StyleGAN2 [23], and GAN Idinvert [48].”
PIE (2020, result): “Very recently, Zhu et al. [2020] introduce a domain-guided embedding method which allows for higher-quality editing, compared to Image2StyleGAN.”
From Continuity to Editability: Inverting GANs with Consecutive Images (2021, methodology): “This problem is addressed by In-domain inversion [24], but it also sacrifices the reconstruction quality.”
Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation (2020, methodology): “latent codes [44], [45], or a hybrid method of them [15], [46].”