Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set

By Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, Xin Tong. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019

887 citations recorded by the source corpus.

Recently, deep learning based 3D face reconstruction methods have shown promising results in both quality and efficiency. However, training deep neural networks typically requires a large volume of data, whereas face images with ground-truth 3D face shapes are scarce. In this paper, we propose a novel deep 3D face reconstruction approach that 1) leverages a robust, hybrid loss function for weakly-supervised learning which takes into account both low-level and perception-level information for supervision, and 2) performs multi-image face reconstruction by exploiting complementary information from different images for shape aggregation. Our method is fast, accurate, and robust to occlusion and large pose. We provide comprehensive experiments on MICC Florence and Facewarehouse datasets, systematically comparing our method with fifteen recent methods and demonstrating its state-of-the-art performance. Code available at https://github.com/Microsoft/Deep3DFaceReconstruction

Subfields: Computer Vision and Pattern Recognition, Dermatology

View Paper: https://www.semanticscholar.org/paper/c5651aea43997f71891c2cc7694ddf51af95c2c0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Dermatology

Citations in the corpus

One Shot Face Swapping on Megapixels (2021, methodology): “As for pose and expression errors, an open-sourced pose estimator [47] and a 3D facial model [13] are used to extract pose and expression vectors.”
I M Avatar: Implicit Morphable Head Avatars from Videos (2021, methodology): “Conventional methods [15, 17, 19, 20, 41, 43, 45, 47] based on morphable mesh models [3, 27, 36] can fit the shape and texture parameters to images for a given subject.”
Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning (2020, methodology): “For these images, we re-estimate ui and u{j} using the 3D reconstruction network [9] (for identity we use a face recognition network [50] to extract deep identity feature instead).”
HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, methodology): “Specifically, we set the dimensionality of the latent codes of HeadNeRF to be the same with the dimensionality of the corresponding codes from 3DMM [51] and initialize them by solving inverse rendering optimization [10, 54] based on the 3DMM model.”
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (2021, methodology): “On the other hand, 3D tools [2, 19, 5, 29] However, their methods cannot render good quality under the “one-shot” condition.”
Learning an animatable detailed 3D face model from in-the-wild images (2020, methodology): “88 Deng et al.19 [2019] 1.”
3D Morphable Face Models—Past, Present, and Future (2019, positive): “[Deng et al. 2019; Sanyal et al. 2019] extend the network architecture to allow for training using multiple images of a person as constraint.”
Cross-Domain and Disentangled Face Manipulation With 3D Guidance (2021, methodology): “the 3D parameters P from the image I using the off-theshelf 3D face reconstruction method [49].”
RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion (2022, background): “Note that our goal is different from face/head reconstruction [13, 51], but to conveniently produce a personalized digital avatar for users.”
Towards Fast, Accurate and Stable 3D Dense Face Alignment (2020, methodology): “In 3D face reconstruction [16,15,47,46,20], the 2D sparse landmarks after projecting are usually used as an extra regularization to facilitate the parameters”
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis (2021, methodology): “Based on the developed 3D face reconstruction [19, 11, 50] and generative adversarial networks, more and more approaches are proposed by intermediately estimating 3D face shapes [22, 43, 55] or facial landmarks [56, 47].”
Efficient Geometry-aware 3D Generative Adversarial Networks (2021, methodology): “We evaluate shape quality by calculating MSE against pseudo-ground-truth depth-maps (Depth) and poses (Pose) estimated from synthesized images by [10]; a similar evaluation was introduced by [59].”
EMOCA: Emotion Driven Monocular Face Capture and Animation (2022, methodology): “Multiple images of the same person, however, are required to train current state-of-the-art 3D face reconstruction methods [19, 27, 73].”