Pivotal Tuning for Latent-based Editing of Real Images

By Daniel Roich, Ron Mokady, Amit H. Bermano, Daniel Cohen-Or. 2021

626 citations recorded by the source corpus.

Recently, numerous facial editing techniques have been proposed that leverage the generative power of a pretrained StyleGAN. To successfully edit an image this way, one must first project (or invert) the image into the pretrained generator’s domain. As it turns out, StyleGAN’s latent space induces an inherent tradeoff between distortion and editability, i.e., between maintaining the original appearance and convincingly altering its attributes. Hence, it remains challenging to apply ID-preserving edits to real facial images. In this article, we present an approach to bridge this gap. The idea is Pivotal Tuning—a brief training process that preserves editing quality, while surgically changing the portrayed identity and appearance. In Pivotal Tuning Inversion, an initial inverted latent code serves as a pivot, around which the generator is fine-tuned. At the same time, a regularization term keeps nearby identities intact, to locally contain the effect. We further show that Pivotal Tuning also applies to accommodating for a multitude of faces, while introducing negligible distortion on the rest of the domain. We validate our technique through inversion and editing metrics and show preferable scores to state-of-the-art methods. Last, we present successful editing for harder cases, including elaborate make-up or headwear.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GAN Inversion: A Survey (2021, neutral): “[65] develop a generator-tuning technique.”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, methodology): “Combining PTI with StyleGAN-XL allows us to invert both in-domain (ImageNet validation set) and out-of-domain images almost precisely.”
Ablating Concepts in Text-to-Image Diffusion Models (2023, background): “Fine-tuning aims to adapt the weights of a pretrained generative model to new domains [73, 46, 72, 41, 79, 34, 47, 80, 30, 35, 24, 44], downstream tasks [71, 54, 78], and test images [6, 53, 48, 31, 25, 49].”
SDEdit: Image Synthesis and Editing with Stochastic Differential Equations (2021, methodology): “Several methods have been proposed in this direction, including ﬁne-tuning network weights for each image [7, 38, 46], choosing better or multiple layers to project and edit [1, 2, 19, 60], designing better encoders [45, 57], modeling image corruption and transformations [5, 24], and discovering…”
Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold (2023, methodology): “5 shows an example, where we apply PTI inversion [Roich et al. 2022] to the real image and then perform a series of manipulations to edit the pose, hair, shape, and expression of the face in the image.”
HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing (2022, background): “0 Model Source License StyleGAN2 [11] Nvidia Source Code License-NC pSp [20] MIT License e4e [26] MIT License ReStyle [1] MIT License PTI [21] MIT License IDInvert [30] MIT License InterFaceGAN [24] MIT License StyleCLIP [18] MIT License GANSpace [6] Apache 2.”
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2022, methodology): “There, it is typical to delicately tune a model to better reconstruct specific faces or scenes (Bau et al., 2019; Roich et al., 2021; Alaluf et al., 2021; Nitzan et al., 2022).”
Zero-shot Image-to-Image Translation (2023, background): “Numerous prior works propose to finetune the GAN model to better match the input image[Bau et al. 2019; Pan et al. 2021; Roich et al. 2021], explore different latent spaces [Abdal et al.”
High-fidelity GAN Inversion with Padding Space (2022, background): “Some also [3,10,38] try to fine-tune the pre-trained generator, leading to better inversion but much slower speed.”
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, methodology): “Several methods have been proposed in this direction, including fine-tuning network weights for each image (Bau et al., 2019a; Pan et al., 2020; Roich et al., 2021), choosing better or multiple layers to project and edit (Abdal et al., 2019; 2020; Gu et al., 2020; Wu et al., 2021), designing better…”
HyperInverter: Improving StyleGAN Inversion via Hypernetwork (2021, methodology): “For other two-stage methods, we compare with PTI [38].”
Prompt-to-Prompt Image Editing with Cross Attention Control (2022, background): “, [51, 1, 3, 35, 50, 43, 45, 47], but has not yet been fully addressed for text-guided diffusion models.”
Paint by Example: Exemplar-based Image Editing with Diffusion Models (2022, background): “Tremendous efforts [2, 5, 36, 49, 55, 57] have been made along this way, mostly relying on the semantic latent space of generative models, e.g ., GANs [16,28,70], yet the majority of existing works are limited to speciﬁc image genres.”
Imagic: Text-Based Real Image Editing with Diffusion Models (2022, methodology): “Similar to GANbased approaches [43,49,64], we begin by finding meaningful representation which, when fed through the generative process, yields images similar to the input image.”
BlobGAN: Spatially Disentangled Scene Representations (2022, methodology): “While this method leads to uneditable, off-manifold latents in previous work [68], we find our blob representation to be more robust in this sense and amenable to naïve optimization.”
High-Fidelity GAN Inversion for Image Attribute Editing (2021, methodology): “We also compare our method with optimization-based methods [1, 20, 27] in Fig.”
Re-Imagen: Retrieval-Augmented Text-to-Image Generator (2022, methodology): “Previously, GANs (Goodfellow et al., 2014) have been used to achieve signiﬁcant performance on image editing (Zhu et al., 2016; Abdal et al., 2019; Zhu et al., 2020; Roich et al., 2021; Tov et al., 2021; Wang et al., 2022; Alaluf et al., 2022).”
Efficient Geometry-aware 3D Generative Adversarial Networks (2021, methodology): “We use pivotal tuning inversion (PTI) [57] to ﬁt test images.”