Unsupervised Training for 3D Morphable Model Regression

By Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sarna, Daniel Vlasic, William T. Freeman. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

326 citations recorded by the source corpus.

We present a method for training a regression network from image pixels to 3D morphable model coordinates using only unlabeled photographs. The training loss is based on features from a facial recognition network, computed on-the-fly by rendering the predicted faces with a differentiable renderer. To make training from features feasible and avoid network fooling effects, we introduce three objectives: a batch distribution loss that encourages the output distribution to match the distribution of the morphable model, a loopback loss that ensures the network can correctly reinterpret its own output, and a multi-view identity loss that compares the features of the predicted 3D face and the input photograph from multiple viewing angles. We train a regression network using these objectives, a set of unlabeled photographs, and the morphable model itself, and demonstrate state-of-the-art results.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/643d11703569766bed0a994941ae5f7b3e101659

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Single-Image Portrait Relighting (2019, background): “Inverse Rendering of Portrait Images Starting with the 3D Morphable Model (3DMM) [6], many inverse rendering methods for portrait images have been proposed [31, 3, 7, 26, 30, 8, 22, 29, 33].”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, background): “Genova et al. [130] address the lack of training data by including randomly generated synthetic faces in each training batch to provide ground truth 3D coordinates, but train the network on real photographs at the same time.”
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision (2019, methodology): “[47] propose OpenDR which approximates the backward pass of the traditional mesh-based graphics pipeline and has inspired several follow-up works [11, 21, 27, 28, 33, 44, 88].”
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set (2019, methodology): “To circumvent these issues, methods have been proposed to train networks without shape labels in an unsupervised or weakly-supervised fashion and promising results have been obtained [49, 48, 53, 46, 16].”
Learning to Infer Implicit Surfaces without 3D Supervision (2019, background): “Aside from silhouettes, shading and appearances in image space also provides supervision cues for learning fine-grained shape representations in category specific domains such as 3D face reconstruction [31, 32, 33, 34, 35] and material inference [36, 37, 38].”
Cross-Domain and Disentangled Face Manipulation With 3D Guidance (2021, background): “With the strong parameterization capability of 3D morphable models (3DMMs) [1], [2], [3], [4], there arises the possibility of editing various facial attributes, including pose, expression, shape,”
Mixture of volumetric primitives for efficient neural rendering (2021, methodology): “models [Genova et al. 2018; Luan Tran 2019; Tewari et al. 2018] from 2D photo and video collections.”
EMOCA: Emotion Driven Monocular Face Capture and Animation (2022, background): “We found that the emotion recognition loss is more difficult to optimize and it requires more careful weighting of the loss compared to the identity recognition losses used in previous work [19, 27, 30].”
Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning (2019, background): “Recent advances in 3D face reconstruction [40, 42, 41, 43, 9], material inference [28, 7] and other 3D reconstruction tasks [51, 38, 34, 14, 23, 35, 39] have leveraged some other forms of differentiable rendering layers to obtain gradient flows in the neural networks.”
GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction (2019, methodology): “4 compares our results with the most recent face reconstruction studies [37, 36, 16, 39, 40] on a subset of MoFA test-set.”
Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance (2020, background): “Differentiable rendering systems for learning geometry comes (mostly) in two ﬂavors: differentiable rasterization [28, 20, 10, 25, 4], and differentiable ray casting.”
3D Morphable Face Models—Past, Present, and Future (2019, unknown): “fferent skin types [Tewari et al. 2018]. This is somewhat improved by discriminative methods that do not need to synthesize the same appearance as a given image, only an image with the same identity [Genova et al. 2018], thereby sidestepping explicit estimation of illumination and camera parameters.”
RetinaFace: Single-stage Dense Face Localisation in the Wild (2019, methodology): “After we predict the shape and texture parameters P ST ∈ R 128 , we employ an efﬁcient differentiable 3D mesh renderer [14] to project a coloured-mesh D P ST onto a 2D image plane with camera parameters e . camera location, camera pose and focal length) and illumination parameters”
Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction (2019, methodology): “In particular, many learning-based techniques [57, 25, 37, 46, 45, 6, 21, 31, 10] have incorporated a differentiable rendering layer to enable an end-to-end architecture for 3D reconstruction and material inference in an unsupervised manner.”
Learning to Regress 3D Face Shape and Expression From an Image Without 3D Supervision (2019, methodology): “[12] propose an end-to-end learning approach using a differentiable rendering process.”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, background): “multiple views of the same object [45], videos [29] or some pre-existing 3D mesh representation that is the starting point for further disentanglement [13, 35, 53, 40] or self-supervision [56].”
Learning an animatable detailed 3D face model from in-the-wild images (2020, methodology): “…and Vetter 1999; Gerig et al. 2018; Romdhani and Vetter 2005; Thies et al. 2016], or learning-based methods [Chang et al. 2018; Deng et al. 2019; Genova et al. 2018; Kim et al. 2018b; Ploumpis et al. 2020; Richardson et al. 2016; Sanyal et al. 2019; Tewari et al. 2017; Tran et al. 2017; Tu et…”