End-to-End 3D Face Reconstruction with Deep Neural Networks

By Pengfei Dou, Shishir K. Shah, Ioannis A. Kakadiaris. 2017

273 citations recorded by the source corpus.

Monocular 3D facial shape reconstruction from a single 2D facial image has been an active research area due to its wide applications. Inspired by the success of deep neural networks (DNN), we propose a DNN-based approach for End-to-End 3D FAce Reconstruction (UH-E2FAR) from a single 2D image. Different from recent works that reconstruct and refine the 3D face in an iterative manner using both an RGB image and an initial 3D facial shape rendering, our DNN model is end-to-end, and thus the complicated 3D rendering process can be avoided. Moreover, we integrate in the DNN architecture two components, namely a multi-task loss function and a fusion convolutional neural network (CNN) to improve facial expression reconstruction. With the multi-task loss function, 3D face reconstruction is divided into neutral 3D facial shape reconstruction and expressive 3D facial shape reconstruction. The neutral 3D facial shape is class-specific. Therefore, higher layer features are useful. In comparison, the expressive 3D facial shape favors lower or intermediate layer features. With the fusion-CNN, features from different intermediate layers are fused and transformed for predicting the 3D expressive facial shape. Through extensive experiments, we demonstrate the superiority of our end-to-end framework in improving the accuracy of 3D face reconstruction.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Dermatology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Dermatology

Citations in the corpus

Extreme 3D Face Reconstruction: Looking Past Occlusions (2017, methodology): “Deep methods were also recently proposed for this purpose [7, 11, 23, 45, 46].”
Modeling Facial Geometry Using Compositional VAEs (2018, background): “With the advent of Deep Learning, there have been several attempts at using deep nets for data-driven face reconstruction [35, 11, 29].”
FaceScape: A Large-Scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction (2020, methodology): “The emergence of 3DMM has sim-pliﬁed the single-view reconstruction of face to a model ﬁtting problem, which could be well solved by ﬁtting facial landmarks and other features[38, 46] or regressing the parameter of 3DMM with a deep neural network[20, 63].”
3D Object Dense Reconstruction from a Single Depth View (2018, background): “For example, morphable 3D models are exploited for face recovery [49] [50].”
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set (2019, methodology): “Some of them apply CNNs to regress 3DMM coefﬁcients [37, 13, 2, 49, 51, 16], some use multi-step schemes to add correction or details onto coarse 3DMM predictions [38, 48, 52, 18], while others advocate direct model-free reconstruction [45, 53, 46, 14].”
Extreme 3D Face Reconstruction: Seeing Through Occlusions (2017, methodology): “Deep methods were also recently proposed for this purpose [6, 11, 24, 49, 51].”
InverseFaceNet: Deep Monocular Inverse Face Rendering (2017, methodology): “Recent techniques approach monocular face reconstruction by fitting active appearance models [1, 17], blendshape models [9, 18, 19, 61], affine face models [15, 16, 20, 46, 51, 54, 58, 62], mesh geometry [26, 33, 47, 48, 52], or volumetric geometry [24] to input images or videos.”
Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks (2017, background): “nose tip or eye centre, which provides rich geometric information for other face analysis tasks such as face recognition [55, 40, 37, 67], emotion estimation [69, 16, 57, 35] and 3D face reconstruction [15, 31, 26, 25, 48, 33, 19].”
Deep Face Recognition: A Survey (2018, methodology): “[48] used a multi-task CNN to divide 3D face reconstruction into neutral 3D reconstruction and expressive 3D reconstruction.”
Learning an animatable detailed 3D face model from in-the-wild images (2020, methodology): “Severalworks are model-free and directly regress 3D faces (i.e. voxels [Jackson et al. 2017] or meshes [Dou et al. 2017; Feng et al. 2018b; Güler et al. 2017; Wei et al. 2019]) and hence can capture more variation than the model-based methods.”
Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression (2019, background): “Face alignment plays an essential role in many face related applications such as face recognition [55, 43, 39, 70, 11], face frontalization [24, 60, 30] and 3D face reconstruction [16, 52, 37, 21].”
Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network (2018, methodology): “[30,31,32,33] propose end-to-end CNN architectures to directly estimate the 3DMM shape parameters.”
3D Object Reconstruction from a Single Depth View with Adversarial Learning (2017, background): “For example, morphable 3D models are exploited for face recovery [3] [9].”
Text-based editing of talking-head video (2019, background): “In the resulting video, the actress appears to be saying fox , even though that word was never spoken by her in the original recording. to directly regress the model parameters [Dou et al. 2017; Genova et al. 2018; Richardson et al. 2016; Tewari et al. 2018a, 2017; Tran et al. 2017].”
EMOCA: Emotion Driven Monocular Face Capture and Animation (2022, methodology): “Model-free approaches directly regress 3D meshes [18,20,28,33,40,70,74,79,93,95,97] or voxels [38] from an image, or optimize a Signed Distance Function (SDF) [61] to fit a face image.”