3D Face Reconstruction by Learning from Synthetic Data

By Elad Richardson, Matan Sela, Ron Kimmel. 2016

319 citations recorded by the source corpus.

Fast and robust three-dimensional reconstruction of facial geometric structure from a single image is a challenging task with numerous applications. Here, we introduce a learning-based approach for reconstructing a three-dimensional face from a single image. Recent face recovery methods rely on accurate localization of key characteristic points. In contrast, the proposed approach is based on a Convolutional-Neural-Network (CNN) which extracts the face geometry directly from its image. Although such deep architectures outperform other models in complex computer vision problems, training them properly requires a large dataset of annotated examples. In the case of three-dimensional faces, currently, there are no large volume data sets, while acquiring such big-data is a tedious task. As an alternative, we propose to generate random, yet nearly photo-realistic, facial images for which the geometric form is known. The suggested model successfully recovers facial shapes from real images, even for faces with extreme expressions and under various lighting conditions.

Subfields: Computer Vision and Pattern Recognition, Neurology, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Neurology

In Signal Processing

Citations in the corpus

A Hybrid Model for Identity Obfuscation by Face Replacement (2018, background): “he-wild images [12,13]. Such parametric models can act as strong regularizers for 3D face reconstruction problems, and have been widely used in optimization based [5,14,12,15,16,17] and learningbased [18,19,20,21,22,23] settings. Recently, a model-based face autoencoder (MoFA) has been introduced [4”
Learning Detailed Face Reconstruction from a Single Image (2016, methodology): “In this setup, common schemes can be divided to 3D morphable model (3DMM) techniques [3, 4], template-based methods [20, 15] and data-driven approaches [26, 41, 34].”
InverseFaceNet: Deep Single-Shot Inverse Face Rendering From A Single Image (2017, background): “In contrast to Richard-son et al. [35, 36], our approach also obtains an estimate of the colored skin reﬂectance and illumination.”
Regressing Robust and Discriminative 3D Morphable Models with a Very Deep Neural Network (2016, methodology): “But despite their success, apart from [31], we are unaware of published reports of using CNNs for 3DMM parameter regression.”
Fake it till you make it: face analysis in the wild using synthetic data alone (2021, background): “Previous work has focused on parts of the face such as the eye region [62] or the hockey mask [45, 76].”
Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network (2018, methodology): “Thus, some works start to use CNNs to estimate the 3D Morphable Model (3DMM) coeﬃcients [10,11,12,13,14,15] or 3D model warping functions [16,17] to restore the corresponding 3D information from a single 2D facial image, which provides both dense face alignment and 3D face reconstruction results.”
Learning an animatable detailed 3D face model from in-the-wild images (2020, methodology): “…1999; Gerig et al. 2018; Romdhani and Vetter 2005; Thies et al. 2016], or learning-based methods [Chang et al. 2018; Deng et al. 2019; Genova et al. 2018; Kim et al. 2018b; Ploumpis et al. 2020; Richardson et al. 2016; Sanyal et al. 2019; Tewari et al. 2017; Tran et al. 2017; Tu et al. 2019].”
Modeling Facial Geometry Using Compositional VAEs (2018, methodology): “Similarly, in [28], used a large dataset of artificially rendered face images to train a CNN that maps images into the space of facial geometry.”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “Richardson et al. [128] used a similar architecture but perform the reconstruction iteratively.”
InverseFaceNet: Deep Monocular Inverse Face Rendering (2017, methodology): “Recent techniques approach monocular face reconstruction by fitting active appearance models [1, 17], blendshape models [9, 18, 19, 61], affine face models [15, 16, 20, 46, 51, 54, 58, 62], mesh geometry [26, 33, 47, 48, 52], or volumetric geometry [24] to input images or videos.”
Text-based editing of talking-head video (2019, background): “to directly regress the model parameters [Dou et al. 2017; Genova et al. 2018; Richardson et al. 2016; Tewari et al. 2018a, 2017; Tran et al. 2017].”
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set (2019, methodology): “Comparison with other methods [37, 53].”
3D Morphable Face Models—Past, Present, and Future (2019, unknown): “Currently, many approaches [Feng et al. 2018; Kim et al. 2018b; Klaudiny et al. 2017; McDonagh et al. 2016; Richardson et al. 2016; Sela et al. 2017; Yu et al. 2017] in the research community are trained on synthetic training data, since it is easy to acquire and comes by design with perfect…”
Extreme 3D Face Reconstruction: Looking Past Occlusions (2017, background): “In one, rather shallow networks are trained on synthetically produced face shapes [35].”
End-to-End 3D Face Reconstruction with Deep Neural Networks (2017, methodology): “Compared with [33, 11, 25], one major difference of our deep neural network is that it is end-to-end and takes only a single RGB image as input.”
MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction (2017, methodology): “Unlike [41, 42], our network is trained completely unsupervised on real images, while they use a synthetic training corpus that lacks realistic features.”
GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction (2019, methodology): “The first such methods either trained regression DCNNs from image to the parameters of a 3DMM [39] or used a 3DMM to synthesize images [28, 18] and formulate an image-to-image translation problem using DCNNs to estimate the depth3 [34].”
Deep video portraits (2018, background): “Learning-based approaches leverage a large corpus of images or image patches to learn a regressor for predicting either 3D face shape and appearance [Richardson et al. 2016; Tewari et al. 2017; Tran et al. 2017], ine-scale skin details [Cao et al. 2015], or both [Richardson et al. 2017; Sela et al.…”
Deep Face Recognition: A Survey (2018, methodology): “[49] employed an iterative 3D CNN by using a secondary input channel to represent the previous network’s output as an image for reconstructing a 3D face as shown in Fig.”