GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction

By Baris Gecer, Stylianos Ploumpis, Irene Kotsia, Stefanos P. Zafeiriou. Computer Vision and Pattern Recognition, 2019

368 citations recorded by the source corpus.

In the past few years, a lot of work has been done towards reconstructing the 3D facial structure from single images by capitalizing on the power of Deep Convolutional Neural Networks (DCNNs). In the most recent works, differentiable renderers were employed in order to learn the relationship between the facial identity features and the parameters of a 3D morphable model for shape and texture. The texture features either correspond to components of a linear texture space or are learned by auto-encoders directly from in-the-wild images. In all cases, the quality of the facial texture reconstruction of the state-of-the-art methods is still not capable of modeling textures in high fidelity. In this paper, we take a radically different approach and harness the power of Generative Adversarial Networks (GANs) and DCNNs in order to reconstruct the facial texture and shape from single images. That is, we utilize GANs to train a very powerful generator of facial texture in UV space. Then, we revisit the original 3D Morphable Models (3DMMs) fitting approaches making use of non-linear optimization to find the optimal latent parameters that best reconstruct the test image but under a new perspective. We optimize the parameters with the supervision of pretrained deep identity features through our end-to-end differentiable framework. We demonstrate excellent results in photorealistic and identity preserving 3D face reconstructions and achieve for the first time, to the best of our knowledge, facial texture reconstruction with high-frequency details.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6220cac2e0fbebea5c28d9e8b7d399eb5c375357

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning an animatable detailed 3D face model from in-the-wild images (2020, methodology): “Instead of capturing high-frequency geometric details, somemethods reconstruct coarse facial geometry along with high-fidelity textures [Gecer et al. 2019; Saito et al. 2017; Slossberg et al. 2018; Yamaguchi et al. 2018].”
Towards Real-World Blind Face Restoration with Generative Facial Prior (2021, background): “We believe the generative facial priors also incorporate conventional geometric priors [9, 70] and even 3D priors [17] for restoration and manipulation, which will be left as a future work.”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, background): “Face generation [19], [227], [228], [229], [230], [231], [232], [233] is relatively easy because the problem includes only one class of objects.”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, background): “CGAN [136] for portrait editing; Vital [137] and Sample-Level GAN (S-GAN) [138] for object tracking; Semi-Latent GAN (SLGAN) [139], Semantically Decomposing GAN (SD-GAN) [140], Wav2Pix [141], GAN-Fit [142], Deforming Autoencoders (DAE) [143], Dual Variational Generator (DVG) [144] and 3D Aided Duet ”
3D Morphable Face Models—Past, Present, and Future (2019, positive): “e original paper [Blanz and Vetter 1999], the first publicly available Morphable Model [Paysan? et al. 2009a], and state-of-the-art facial re-enactment results [Kim et al. 2018] and GAN-based models [Gecer et al. 2019]. In this paper, we provide a detailed survey of 3D Morphable Face Models over the”
I M Avatar: Implicit Morphable Head Avatars from Videos (2021, methodology): “Conventional methods [15, 17, 19, 20, 41, 43, 45, 47] based on morphable mesh models [3, 27, 36] can fit the shape and texture parameters to images for a given subject.”
Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs (2020, background): “For example, Gecer et al. (2019); Kanazawa et al. (2018a); Sanyal et al. (2019); Shang et al. (2020) reconstruct the 3D shape of human faces or bodies using 3DMM (Blanz & Vetter, 1999) or SMPL (Loper et al., 2015) as a prior knowledge.”
Towards Fast, Accurate and Stable 3D Dense Face Alignment (2020, methodology): “In 3D face reconstruction [16,15,47,46,20], the 2D sparse landmarks after projecting are usually used as an extra regularization to facilitate the parameters”
Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, methodology): “For objects such as human bodies and human faces, some methods [26, 17, 60, 14] have learn to reconstruct from raw images, but starting from the knowledge of a predefined shape model such as SMPL [34] or Basel [43].”