Regressing Robust and Discriminative 3D Morphable Models with a Very Deep Neural Network

By Anh Tuan Tran, Tal Hassner, Iacopo Masi, Gérard G. Medioni. 2016

504 citations recorded by the source corpus.

The 3D shapes of faces are well known to be discriminative. Yet despite this, they are rarely used for face recognition and always under controlled viewing conditions. We claim that this is a symptom of a serious but often overlooked problem with existing methods for single view 3D face reconstruction: when applied in the wild, their 3D estimates are either unstable and change for different photos of the same subject or they are over-regularized and generic. In response, we describe a robust method for regressing discriminative 3D morphable face models (3DMM). We use a convolutional neural network (CNN) to regress 3DMM shape and texture parameters directly from an input photo. We overcome the shortage of training data required for this purpose by offering a method for generating huge numbers of labeled examples. The 3D estimates produced by our CNN surpass state of the art accuracy on the MICC data set. Coupled with a 3D-3D face matching pipeline, we show the first competitive face recognition results on the LFW, YTF and IJB-A benchmarks using 3D face shapes as representations, rather than the opaque deep feature vectors used by other modern systems.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

A Hybrid Model for Identity Obfuscation by Face Replacement (2018, background): “Note that the shape and reflectance parameters α and β of the face mode depend on the identity of the person [10,20].”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “Tran et al. [127] use an iterative optimization to ﬁt an expressionless model to a large number of photographs, and treat the results where the optimization converged as ground truth.”
Extreme 3D Face Reconstruction: Seeing Through Occlusions (2017, methodology): “We used the same face verification system from [51], and we refer to that paper for more details.”
GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction (2019, methodology): “4 compares our results with the most recent face reconstruction studies [37, 36, 16, 39, 40] on a subset of MoFA test-set.”
Nonlinear 3D Face Morphable Model (2018, methodology): “[13], an offline optimization method, while surpassing all other regression methods [30, 35, 36].”
State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications (2018, methodology): “Face shape and texture estimation can be used for face identiﬁcation [TTHMM17].”
Unsupervised Training for 3D Morphable Model Regression (2018, methodology): “[29] are more resistant to confounding variables.”
InverseFaceNet: Deep Monocular Inverse Face Rendering (2017, methodology): “Recent techniques approach monocular face reconstruction by fitting active appearance models [1, 17], blendshape models [9, 18, 19, 61], affine face models [15, 16, 20, 46, 51, 54, 58, 62], mesh geometry [26, 33, 47, 48, 52], or volumetric geometry [24] to input images or videos.”
Towards Fast, Accurate and Stable 3D Dense Face Alignment (2020, background): “Recent studies are mainly divided into two categories: 3D Morphable Model (3DMM) parameters regression [29,59,34,36,48,60,23] and dense vertices regression [28,18].”
SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild' (2017, background): “Previous works have attempted to learn surface normal from synthetic data [35, 28], which often performs imperfectly in the presence of real world variations like illumination and expression.”
Learning an animatable detailed 3D face model from in-the-wild images (2020, methodology): “We compare DECA with publicly available methods, namely 3DDFA-V2 [Guo et al. 2020], Deng et al. [2019], RingNet [Sanyal et al. 2019], PRNet [Feng et al. 2018b], 3DMM-CNN [Tran et al. 2017] and Ex-treme3D [Tran et al. 2018].”
InverseFaceNet: Deep Single-Shot Inverse Face Rendering From A Single Image (2017, methodology): “Recent techniques approach monocular face reconstruction by ﬁtting active appearance models [1, 12], blendshape models [6, 13, 14], multi-linear face models [35, 36, 41, 46], or mesh geometry [19, 25, 37] to input images or videos.”
Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, methodology): “For objects such as human bodies and human faces, some methods [16], [17], [18], [20], [21], [29], [57], [58], [59], [60], [61] have learn to reconstruct from raw images, but starting from the knowledge of a predefined shape model, such as SMPL [62] or Basel [11], or shape templates.”
Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network (2018, background): “[15, 57, 31, 36] propose end-to-end CNN architectures to directly estimate the 3DMM shape parameters.”
Deep Face Recognition: A Survey (2018, methodology): “one to many generating many patches or images of the pose variability from a single image 3D model [109], [107], [129], [130], [48] [57], [155], [154] 2D deep model [231], [221], [141] data augmentation [99], [228], [46] [177], [145], [146], [150], [160]”
Learning to Regress 3D Face Shape and Expression From an Image Without 3D Supervision (2019, methodology): “We evaluate RingNet qualitatively and quantitatively and compare our results with publicly available methods, namely: PRNet (ECCV 2018 [9]), Extreme3D (CVPR 2018 [35]) and 3DMM-CNN (CVPR 2017 [34]).”
Multilinear Autoencoder for 3D Face Model Learning (2018, methodology): “The regression from the 2D heigthmap to the model coefficients is implemented using ResNet-18 [14], a state-of-the-art architecture which has recently shown very good performances in face-related problems [20, 25].”
Deep video portraits (2018, background): “Learning-based approaches leverage a large corpus of images or image patches to learn a regressor for predicting either 3D face shape and appearance [Richardson et al. 2016; Tewari et al. 2017; Tran et al. 2017], ine-scale skin details [Cao et al.”
EMOCA: Emotion Driven Monocular Face Capture and Animation (2022, background): “The latter are trained fully supervised [15, 34, 42, 67, 87, 88, 98] or self-supervised with predicted 2D keypoints [19, 27, 52, 73, 76, 80, 82, 83, 96], 2D face contours [52], photometric constraints [19, 27, 30, 76, 80, 82, 83, 96], face recognition features [19, 27, 30, 76], multi-view constraint”