Self-Supervised Multi-level Face Model Learning for Monocular Reconstruction at Over 250 Hz

By Ayush Kumar Tewari, Michael Zollhöfer, Pablo Garrido, Florian Bernard, Hyeongwoo Kim, Patrick Pérez, Christian Theobalt. 2017

277 citations recorded by the source corpus.

The reconstruction of dense 3D models of face geometry and appearance from a single image is highly challenging and ill-posed. To constrain the problem, many approaches rely on strong priors, such as parametric face models learned from limited 3D scan data. However, prior models restrict generalization of the true diversity in facial geometry, skin reflectance and illumination. To alleviate this problem, we present the first approach that jointly learns 1) a regressor for face shape, expression, reflectance and illumination on the basis of 2) a concurrently learned parametric face model. Our multi-level face model combines the advantage of 3D Morphable Models for regularization with the out-of-space generalization of a learned corrective space. We train end-to-end on in-the-wild images without dense annotations by fusing a convolutional encoder with a differentiable expert-designed renderer and a self-supervised training loss, both defined at multiple detail levels. Our approach compares favorably to the state-of-the-art in terms of reconstruction quality, better generalizes to real world faces, and runs at over 250 Hz.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

A Hybrid Model for Identity Obfuscation by Face Replacement (2018, methodology): “age to the output of the decoder as E loss(p) = E land(p) + w photoE photo(p) + w regE reg(p): (1) Here, E land(p) is a landmark alignment term which measures the distance between 66 ducial landmarks [13] in the input image with the corresponding landmarks on the output of the parametric decoder, E ”
Extreme 3D Face Reconstruction: Seeing Through Occlusions (2017, methodology): “Contrary to these methods, deep networks were used in [6, 11, 24, 44, 49, 50, 51] to estimate 3D shapes with an emphasis on unconstrained photos.”
Learning to Infer Implicit Surfaces without 3D Supervision (2019, background): “Aside from silhouettes, shading and appearances in image space also provides supervision cues for learning fine-grained shape representations in category specific domains such as 3D face reconstruction [31, 32, 33, 34, 35] and material inference [36, 37, 38].”
Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning (2019, background): “Recent advances in 3D face reconstruction [40, 42, 41, 43, 9], material inference [28, 7] and other 3D reconstruction tasks [51, 38, 34, 14, 23, 35, 39] have leveraged some other forms of differentiable rendering layers to obtain gradient flows in the neural networks.”
3D Morphable Face Models—Past, Present, and Future (2019, unknown): “]. As the problem size increases, as in the case of shape-fromshading, gradient descent becomes the most common optimization approach [Garrido et al. 2016a; Shi et al. 2014; Suwajanakorn et al. 2014; Tewari et al. 2018]. In all the above methods, the discontinuity of the appearance function is dealt”
Mixture of volumetric primitives for efficient neural rendering (2021, methodology): “Differentiable rendering strategies have for example been employed for learning 3D face models [Genova et al. 2018; Luan Tran 2019; Tewari et al. 2018] from 2D photo and video collections.”
Extreme 3D Face Reconstruction: Looking Past Occlusions (2017, methodology): “Contrary to these methods, deep networks were used in [7, 11, 23, 40, 44, 45, 46] to estimate 3D shapes with an emphasis on unconstrained photos.”
Learning to Regress 3D Face Shape and Expression From an Image Without 3D Supervision (2019, methodology): “None of the fitting-based methods produce true ground truth for real world face images, while synthetically generated faces may not generalize well to the real world [31].”
Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction (2019, methodology): “In particular, many learning-based techniques [57, 25, 37, 46, 45, 6, 21, 31, 10] have incorporated a differentiable rendering layer to enable an end-to-end architecture for 3D reconstruction and material inference in an unsupervised manner.”
Tex2Shape: Detailed Full Human Body Geometry From a Single Image (2019, methodology): “Also related to our approach are recent works integrating a differentiable face renderer in a neural network to estimate instance correctives of geometry and albedo relative to a base model [47], or learn an identity geometry and albedo basis from scratch from video [46].”
3D Hand Shape and Pose From Images in the Wild (2019, methodology): “This method could benefit greatly from a hand appearance model by leveraging a photometric loss in training as proposed in [56, 55] for faces.”
GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction (2019, methodology): “The recent 3D face fitting methods [36, 40, 14] still make use of similar statistical models for the texture.”
FaceScape: A Large-Scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction (2020, methodology): “Another direction is to enhance the representation power of 3DMM by using deep neural network to present 3DMM bases [2, 45, 48, 50, 49, 16].”
EMOCA: Emotion Driven Monocular Face Capture and Animation (2022, background): “The latter are trained fully supervised [15, 34, 42, 67, 87, 88, 98] or self-supervised with predicted 2D keypoints [19, 27, 52, 73, 76, 80, 82, 83, 96], 2D face contours [52], photometric constraints [19, 27, 30, 76, 80, 82, 83, 96], face recognition features [19, 27, 30, 76], multi-view constraint”
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set (2019, methodology): “To circumvent these issues, methods have been proposed to train networks without shape labels in an unsupervised or weakly-supervised fashion and promising results have been obtained [49, 48, 53, 46, 16].”
Learning an animatable detailed 3D face model from in-the-wild images (2020, background): “[67, 68] jointly learn a statistical face model and reconstruct 3D faces from images.”