StyleRig: Rigging StyleGAN for 3D Control Over Portrait Images

By Ayush Kumar Tewari, Mohamed A. Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt. 2020

451 citations recorded by the source corpus.

StyleGAN generates photorealistic portrait images of faces with eyes, teeth, hair and context (neck, shoulders, background), but lacks a rig-like control over semantic face parameters that are interpretable in 3D, such as face pose, expressions, and scene illumination. Three-dimensional morphable face models (3DMMs) on the other hand offer control over the semantic parameters, but lack photorealism when rendered and only model the face interior, not other parts of a portrait image (hair, mouth interior, background). We present the first method to provide a face rig-like control over a pretrained and fixed StyleGAN via a 3DMM. A new rigging network, \textit{RigNet} is trained between the 3DMM's semantic parameters and StyleGAN's input. The network is trained in a self-supervised manner, without the need for manual annotations. At test time, our method generates portrait images with the photorealism of StyleGAN and provides explicit control over the 3D semantic parameters of the face.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, methodology): “Many works explore how to utilize the latent space of a pretrained generator for image manipulation [6, 45, 50].”
GAN Inversion: A Survey (2021, methodology): “The recent StyleRig method [102] is trained to align the parameters of the 3D morphable model (3DMM) [179] with the input of StyleGAN [13].”
Designing an encoder for StyleGAN image manipulation (2021, methodology): “[37] use a pretrained 3DMM to edit expression, pose and illumination of faces by borrowing them from other latent codes.”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, methodology): “[36] utilize a pretrained 3DMM to learn semantic face edits in the latent space.”
Third Time's the Charm? Image and Video Editing with StyleGAN3 (2022, background): “StyleGAN2 [36] features several semantically-rich latent spaces, which have been heavily studied and exploited in the context of image manipulation [4,15,26,60,61,65] and image inversion [1,2,10,25,51,85].”
Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs (2020, background): “Recent works further indicate that various semantic meanings may be implicitly encoded in the latent space of pre-trained GANs even though the concepts are not specified in the training set, ranging from memorability [13], transformation [18], to facial attributes [25, 12, 27].”
HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, background): “Some strategies, such as adding disentangled loss [9, 26, 49], embedding the above-mentioned parametric face model into generative adversarial network (GAN) [5, 14], etc.”
Improved StyleGAN Embedding: Where are the Good Latents? (2020, methodology): “In this work, we measure the reconstruction quality of an embedding both i) absolutely using RMSE, PSNR and ii) perceptually using SSIM, VGG perceptual similarity, LPIPS perceptual similarity and the FID [12] score between the input and embedded image.”
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation (2021, background): “While 2D latent space manipulation produces realistic results, these methods tend to lack explicit camera control, have no 3D understanding, require shape priors from 3DMM models [67, 68], or reconstruct the surface as a preprocessing step [42].”
SofGAN: A Portrait Image Generator with Dynamic Styling (2020, methodology): “The seminal attribute-based approaches [Abdal et al. 2020; Chen et al. 2016; Shen et al. 2020; Tewari et al. 2020] explore to find attribute-specific paths in the latent spaces, e.g., to use direction codes to drag random style code towards each attribute direction.”
Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold (2023, methodology): “Most previous approaches gain controllability of GANs via prior 3D models [Deng et al. 2020; Ghosh et al. 2020; Tewari et al. 2020] or supervised learning that relies on manually annotated data [Abdal et al. 2021; Isola et al. 2017; Ling et al. 2021; Park et al. 2019; Shen et al. 2020].”
PIE (2020, methodology): “We augment the synthesis term with an editing energy that is based on the StyleRig framework [Tewari et al. 2020], which allows us to obtain more accurate semantic editing while preserving the non-edited attributes.”
Disentangled Face Attribute Editing via Instance-Aware Latent Space Search (2021, background): “Recently, 3D prior ( e.g. 3DMM [Blanz and Vetter, 1999]) is also introduced to encode the synthetic image [Deng et al. , 2020; Tewari et al. , 2020].”
Style Intervention: How to Achieve Spatial Disentanglement with Style-based Generators? (2020, methodology): “Supervised methods [3, 5, 21, 25], on the other hand, edit latent codes under the guidance of attribute labels in certain translation tasks.”
StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis (2021, background): “Some meth-ods (H¨ark¨onen et al., 2020; Tewari et al., 2020a; Shen et al., 2020; Abdal et al., 2020; Tewari et al., 2020b; Leimk¨uhler & Drettakis, 2021; Shoshan et al., 2021) leverage disentangled properties in the latent space to enable explicit controls, most of which focus on faces.”
HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing (2021, background): “These range from full-supervision in the form of semantic labels [3, 16, 19, 56] and facial priors [59,60] to unsupervised approaches [24,57,63,64].”