PIE

By Ayush Kumar Tewari, Mohamed A. Elgharib, Mallikarjun B R, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt. 2020

52 citations recorded by the source corpus.

Editing of portrait images is a very popular and important research topic with a large variety of applications. For ease of use, control should be provided via a semantically meaningful parameterization that is akin to computer animation controls. The vast majority of existing techniques do not provide such intuitive and fine-grained control, or only enable coarse editing of a single isolated control parameter. Very recently, high-quality semantically controlled editing has been demonstrated, however only on synthetically created StyleGAN images. We present the first approach for embedding real portrait images in the latent space of StyleGAN, which allows for intuitive editing of the head pose, facial expression, and scene illumination in the image. Semantic editing in parameter space is achieved based on StyleRig, a pretrained neural network that maps the control space of a 3D morphable face model to the latent space of the GAN. We design a novel hierarchical non-linear optimization problem to obtain the embedding. An identity preservation energy term allows spatially coherent edits while maintaining facial integrity. Our approach runs at interactive frame rates and thus allows the user to explore the space of possible edits. We evaluate our approach on a wide set of portrait photos, compare it to the current state of the art, and validate the effectiveness of its components in an ablation study.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

GAN Inversion: A Survey (2021, neutral): “3 7 7 7 7 [14], [15] [14], [33], [44] PIE [100] 2020, TOG O.”
AgileGAN: Stylizing Portraits by Inversion-Consistent Transfer Learning (2021, background): “One approach is based on optimization [Abdal et al. 2019a; Karras et al. 2019; Tewari et al. 2020], directly optimizing the latent code to minimize the pixel-wise reconstruction loss for a single input instance.”
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation (2021, background): “While 2D latent space manipulation produces realistic results, these methods tend to lack explicit camera control, have no 3D understanding, require shape priors from 3DMM models [64, 65], or reconstruct the surface as a preprocessing step [40].”
AgileGAN (2021, background): “One approach is based on optimization [Abdal et al. 2019a; Karras et al. 2019; Tewari et al. 2020], directly optimizing the latent code to minimize the pixel-wise reconstruction loss for a single input instance.”
Only a matter of style: age transformation using a style-based regression model (2021, background): “Recent years have seen diverse approaches for performing such an inversion [Abdal et al. 2019, 2020a; Bau et al. 2019; Creswell and Bharath 2018; Pidhorskyi et al. 2020; Richardson et al. 2020; Tewari et al. 2020; Tov et al. 2021; Xia et al. 2021; Zhu et al. 2020].”
HyperInverter: Improving StyleGAN Inversion via Hypernetwork (2021, methodology): “PIE [43] opts ﬁrst to use an optimization process to locate the latent code in W space to preserve the editing ability and then optimize further the latent in the W + space to enhance the reconstruction quality.”
Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks (2021, methodology): “Another framework, StyleRig (Tewari et al., 2020a), maps the latent space of the GAN to a 3D model.”
StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis (2021, background): “Some meth-ods (H¨ark¨onen et al., 2020; Tewari et al., 2020a; Shen et al., 2020; Abdal et al., 2020; Tewari et al., 2020b; Leimk¨uhler & Drettakis, 2021; Shoshan et al., 2021) leverage disentangled properties in the latent space to enable explicit controls, most of which focus on faces.”
Pivotal Tuning for Latent-based Editing of Real Images (2021, methodology): “Tov et al. [2021] define this as the distortion-editability tradeoff and design an encoder that attempts to find a “sweet-spot” in this tradeoff. Similarly, the tradeoff was also demonstrated by Zhu et al. [2020], who suggests an improved embedding algorithm using a novel regularization method.”