MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction

By Ayush Kumar Tewari, Michael Zollhöfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Pérez, Christian Theobalt. 2017

585 citations recorded by the source corpus.

In this work we propose a novel model-based deep convolutional autoencoder that addresses the highly challenging problem of reconstructing a 3D human face from a single in-the-wild color image. To this end, we combine a convolutional encoder network with an expert-designed generative model that serves as decoder. The core innovation is the differentiable parametric decoder that encapsulates image formation analytically based on a generative model. Our decoder takes as input a code vector with exactly defined semantic meaning that encodes detailed face pose, shape, expression, skin reflectance and scene illumination. Due to this new way of combining CNN-based with model-based face reconstruction, the CNN-based encoder learns to extract semantically meaningful parameters from a single monocular input image. For the first time, a CNN encoder and an expert-designed generative model can be trained end-to-end in an unsupervised manner, which renders training on very large (unlabeled) real world data feasible. The obtained reconstructions compare favorably to current state-of-the-art approaches in terms of quality and richness of representation.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Hybrid Model for Identity Obfuscation by Face Replacement (2018, methodology): “Our stage-I architecture is based on the Model-based Face Autoencoder (MoFA) [4] and consists of a convolutional encoder and a parametric face decoder.”
Generating 3D faces using Convolutional Mesh Autoencoders (2018, methodology): “Our main contributions are: 1) we introduce a Convolutional Mesh Autoencoder consisting of mesh downsampling and mesh upsampling layers with fast localized convolutional filters defined on the mesh surface; 2) we show that our model accurately represents 3D faces in a low-dimensional latent space pe”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, background): “An exception is texture inference on human faces [2,22,23,28], but these approaches require a large-set of 3D ground truth data with high quality texture maps.”
Deep Face Recognition: A Survey (2018, background): “one to many generating many patches or images of the pose variability from a single image 3D model [109], [107], [129], [130], [48] [57], [155], [154] 2D deep model [231], [221], [141] data augmentation [99], [228], [46] [177], [145], [146], [150], [160]”
Learning an animatable detailed 3D face model from in-the-wild images (2020, methodology): “…1999; Gerig et al. 2018; Romdhani and Vetter 2005; Thies et al. 2016], or learning-based methods [Chang et al. 2018; Deng et al. 2019; Genova et al. 2018; Kim et al. 2018b; Ploumpis et al. 2020; Richardson et al. 2016; Sanyal et al. 2019; Tewari et al. 2017; Tran et al. 2017; Tu et al. 2019].”
InverseFaceNet: Deep Monocular Inverse Face Rendering (2017, methodology): “Recent techniques approach monocular face reconstruction by fitting active appearance models [1, 17], blendshape models [9, 18, 19, 61], affine face models [15, 16, 20, 46, 51, 54, 58, 62], mesh geometry [26, 33, 47, 48, 52], or volumetric geometry [24] to input images or videos.”
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set (2019, methodology): “To circumvent these issues, methods have been proposed to train networks without shape labels in an unsupervised or weakly-supervised fashion and promising results have been obtained [49, 48, 53, 46, 16].”
Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction (2019, methodology): “In particular, many learning-based techniques [57, 25, 37, 46, 45, 6, 21, 31, 10] have incorporated a differentiable rendering layer to enable an end-to-end architecture for 3D reconstruction and material inference in an unsupervised manner.”
Deep Single-Image Portrait Relighting (2019, methodology): “Inverse Rendering of Portrait Images Starting with the 3D Morphable Model (3DMM) [6], many inverse rendering methods for portrait images have been proposed [31, 3, 7, 26, 30, 8, 22, 29, 33].”
3D Hand Shape and Pose From Images in the Wild (2019, methodology): “This method could benefit greatly from a hand appearance model by leveraging a photometric loss in training as proposed in [56, 55] for faces.”
Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network (2018, methodology): “Unsupervised methods have been also researched recently, [34,35] can regress the 3DMM coeﬃcients without the help of training data by using facial texture information, which performs badly in faces with large-poses and unbalanced illuminations.”
Modeling Facial Geometry Using Compositional VAEs (2018, background): “With the advent of Deep Learning, there have been several attempts at using deep nets for data-driven face reconstruction [35, 11, 29].”
Self-supervised learning of a facial attribute embedding from video (2018, background): “As ﬁtting a 3DMM is relatively expensive, [3, 50] learn this end-to-end using either landmarks or a photometric error as supervision.”
Learning Self-Consistency for Deepfake Detection (2020, methodology): “3D models [44], AutoEncoders [43, 49], or Generative Adversarial Networks [16, 17, 29, 4] are used to generate the fake image segment, which is then blended back to the original image.”
Cross-Domain and Disentangled Face Manipulation With 3D Guidance (2021, background): “differentiable graphics pipelines [38], [39], [40], combining the traditional pipelines with learnable components [41],”
GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction (2019, methodology): “4 compares our results with the most recent face reconstruction studies [37, 36, 16, 39, 40] on a subset of MoFA test-set.”
Unsupervised Training for 3D Morphable Model Regression (2018, methodology): “An approach closely related to ours was recently proposed by Tewari, et al. [28], who train an autoencoder network on unlabeled photographs to predict shape, expression, texture, pose, and lighting simultaneously.”