Designing an encoder for StyleGAN image manipulation

By Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, Daniel Cohen-Or. 2021

908 citations recorded by the source corpus.

Recently, there has been a surge of diverse methods for performing image editing by employing pre-trained unconditional generators. Applying these methods on real images, however, remains a challenge, as it necessarily requires the inversion of the images into their latent space. To successfully invert a real image, one needs to find a latent code that reconstructs the input image accurately, and more importantly, allows for its meaningful manipulation. In this paper, we carefully study the latent space of StyleGAN, the state-of-the-art unconditional generator. We identify and analyze the existence of a distortion-editability tradeoff and a distortion-perception tradeoff within the StyleGAN latent space. We then suggest two principles for designing encoders in a manner that allows one to control the proximity of the inversions to regions that StyleGAN was originally trained on. We present an encoder based on our two principles that is specifically designed for facilitating editing on real images by balancing these tradeoffs. By evaluating its performance qualitatively and quantitatively on numerous challenging domains, including cars and horses, we show that our inversion method, followed by common editing techniques, achieves superior real-image editing quality, with only a small reconstruction accuracy drop.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generative Adversarial Networks (2021, neutral): “One may also use StyleGAN for image editing by calculating the latent vector of a given input image in the styleGAN (this operation is known as styleGAN inversion) and then manipulating this vector for editing the image [75, 60, 1, 68, 2, 65, 64, 54].”
InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “Numerous editing approaches invert [1–3, 12] or encode [8, 50, 63] images into a latent space (e.”
SDEdit: Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “…including ﬁne-tuning network weights for each image [7, 38, 46], choosing better or multiple layers to project and edit [1, 2, 19, 60], designing better encoders [45, 57], modeling image corruption and transformations [5, 24], and discovering meaningful latent directions [49, 16, 27, 21].”
High-Fidelity GAN Inversion for Image Attribute Editing (2021, methodology): “We also calculate the identity loss Lid = 1 − ⟨F (X), F (X̂)⟩, where F is pre-trained ArcFace [8] or a ResNet-50 model for different domains [34].”
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2022, methodology): “Progressive extensions We follow Tov et al. (2021) and consider a progressive multi-vector setup.”
HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing (2022, methodology): “[26].”
Imagic: Text-Based Real Image Editing with Diffusion Models (2022, methodology): “Similar to GANbased approaches [43,49,64], we begin by finding meaningful representation which, when fed through the generative process, yields images similar to the input image.”
High-fidelity GAN Inversion with Padding Space (2022, methodology): “Generative Adversarial Networks (GANs).”
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, methodology): “, 2021), designing better encoders (Richardson et al., 2021; Tov et al., 2021), modeling image corruption and transformations (Anirudh et al.”
StyleAlign: Analysis and Applications of Aligned StyleGAN Models (2021, methodology): “…recent works, either by using: (i) optimization (Abdal et al., 2019; Karras et al., 2020b); or (ii) an encoder (Guan et al., 2020; Pidhorskyi et al., 2020; Richardson et al., 2021; Tov et al., 2021); or (iii) a hybrid approach using both (Zhu et al., 2016; Baylies, 2019; Zhu et al., 2020a).”
Label-Efficient Semantic Segmentation with Diffusion Models (2021, methodology): “• GAN Inversion employs the state-of-the-art method (Tov et al., 2021) to obtain the latent codes for real images.”
DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation (2021, methodology): “To demonstrate the nearly perfect reconstruction performance of our method, we perform the quantitative comparison with SOTA GAN inversion methods, pSp [32], e4e [40], ReStyle [3] and HFGI [43].”
Text2LIVE: Text-Driven Layered Image and Video Editing (2022, methodology): “However, editing a real input image using these methods requires first tackling the GAN-inversion challenge [39,47].”