Nonlinear 3D Face Morphable Model

By Luan Tran, Xiaoming Liu. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

431 citations recorded by the source corpus.

As a classic statistical model of 3D facial shape and texture, 3D Morphable Model (3DMM) is widely used in facial analysis, e.g., model fitting, image synthesis. Conventional 3DMM is learned from a set of well-controlled 2D face images with associated 3D face scans, and represented by two sets of PCA basis functions. Due to the type and amount of training data, as well as the linear bases, the representation power of 3DMM can be limited. To address these problems, this paper proposes an innovative framework to learn a nonlinear 3DMM model from a large set of unconstrained face images, without collecting 3D face scans. Specifically, given a face image as input, a network encoder estimates the projection, shape and texture parameters. Two decoders serve as the nonlinear 3DMM to map from the shape and texture parameters to the 3D shape and texture, respectively. With the projection parameter, 3D shape, and texture, a novel analytically-differentiable rendering layer is designed to reconstruct the original input face. The entire network is end-to-end trainable with only weak supervision. We demonstrate the superior representation power of our nonlinear 3DMM over its linear counterpart, and its contribution to face alignment and 3D reconstruction.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/09ec60f2eea5d43792b2bc9da63b1c9b7719f666

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

RetinaFace: Single-stage Dense Face Localisation in the Wild (2019, background): “Recently, self-supervised 3D mor-phable models [14, 51, 52, 70] have achieved promising 3D face modelling in-the-wild.”
Learning to Regress 3D Face Shape and Expression From an Image Without 3D Supervision (2019, methodology): “Tran and Liu [36] learn a nonlinear 3DMM model by using an analytically differentiable rendering layer and in a weakly supervised fashion with 3D data.”
GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction (2019, methodology): “The recent 3D face fitting methods [36, 40, 14] still make use of similar statistical models for the texture.”
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set (2019, methodology): “To circumvent these issues, methods have been proposed to train networks without shape labels in an unsupervised or weakly-supervised fashion and promising results have been obtained [49, 48, 53, 46, 16].”
HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, methodology): “Specifically, we set the dimensionality of the latent codes of HeadNeRF to be the same with the dimensionality of the corresponding codes from 3DMM [51] and initialize them by solving inverse rendering optimization [10, 54] based on the 3DMM model.”
Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning (2019, background): “Recent advances in 3D face reconstruction [40, 42, 41, 43, 9], material inference [28, 7] and other 3D reconstruction tasks [51, 38, 34, 14, 23, 35, 39] have leveraged some other forms of differentiable rendering layers to obtain gradient flows in the neural networks.”
FaceScape: A Large-Scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction (2020, methodology): “Another direction is to enhance the representation power of 3DMM by using deep neural network to present 3DMM bases [2, 45, 48, 50, 49, 16].”
Learning to Infer Implicit Surfaces without 3D Supervision (2019, background): “Aside from silhouettes, shading and appearances in image space also provides supervision cues for learning fine-grained shape representations in category specific domains such as 3D face reconstruction [31, 32, 33, 34, 35] and material inference [36, 37, 38].”
Deep Single-Image Portrait Relighting (2019, background): “Inverse Rendering of Portrait Images Starting with the 3D Morphable Model (3DMM) [6], many inverse rendering methods for portrait images have been proposed [31, 3, 7, 26, 30, 8, 22, 29, 33].”
Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction (2019, background): “Before the advent of differentiable renderer, a majority of prior works use supervised deep learning approaches, which seek to learn the nonlinear mapping between the input image and 3D model parameters [47, 53, 41, 49] from labeled groundtruth data.”
3D Morphable Face Models—Past, Present, and Future (2019, positive): “This is now very common, including in the current state-of-the-art, e.g., [Tran et al. 2019; Tran and Liu 2018b].”
I M Avatar: Implicit Morphable Head Avatars from Videos (2021, methodology): “Extensions to this idea included multilinear models for shape and expression [6, 55], full-head PCA models [13, 27, 37], deep non-linear models [40, 52] and fully articulated head models with linear blend skinning (LBS) and corrective blendshapes [27].”
A survey on face data augmentation for the training of deep neural networks (2019, methodology): “[95] utilized neural networks to reconstruct nonlinear 3D face morphable model which is a more flexible representation.”
Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs (2020, background): “And Tran & Liu (2018); Kanazawa et al. (2018b) adopt the guidance of 2D key-points to learn the 3D shape of face or objects.”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, background): “Very recently, a few works have started tackling the problem of recovering the three-dimensional geometry of objects from more limited information—[18] use keypoints and masks to learn a 3D deformable model of birds, [44] use keypoints and semisupervised pretraining using a 3DMM, while [50] use a 3D”