Learning to Regress 3D Face Shape and Expression From an Image Without 3D Supervision

By Soubhik Sanyal, Timo Bolkart, Haiwen Feng, Michael J. Black. Computer Vision and Pattern Recognition, 2019

352 citations recorded by the source corpus.

The estimation of 3D face shape from a single image must be robust to variations in lighting, head pose, expression, facial hair, makeup, and occlusions. Robustness requires a large training set of in-the-wild images, which by construction, lack ground truth 3D shape. To train a network without any 2D-to-3D supervision, we present RingNet, which learns to compute 3D face shape from a single image. Our key observation is that an individual’s face shape is constant across images, regardless of expression, pose, lighting, etc. RingNet leverages multiple images of a person and automatically detected 2D face features. It uses a novel loss that encourages the face shape to be similar when the identity is the same and different for different people. We achieve invariance to expression by representing the face using the FLAME model. Once trained, our method takes a single image and outputs the parameters of FLAME, which can be readily animated. Additionally we create a new database of faces “not quite in-the-wild” (NoW) with 3D head scans and high-resolution images of the subjects in a wide variety of conditions. We evaluate publicly available methods and find that RingNet is more accurate than methods that use 3D supervision. The dataset, model, and results are available for research purposes at http://ringnet.is.tuebingen.mpg.de.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/692dc8750e08f16137a5e9445827f0ba921fa8e0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

EMOCA: Emotion Driven Monocular Face Capture and Animation (2022, methodology): “While, for the task of 3D face reconstruction, standard benchmarks exist to quantitatively evaluate the identity face shape [29, 73], no such benchmark exists to assess the accuracy of the reconstructed expression.”
StyleRig: Rigging StyleGAN for 3D Control Over Portrait Images (2020, methodology): “Our approach brings the best of both worlds: the controllable parametric nature of existing morphable face models [26, 31], and the high photorealism of generative face models [17, 18].”
Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, methodology): “3DM, VP, T, E from I Face [46] 3DMM, 2DKP, I Fit 3DMM to I Face”
Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs (2020, background): “For example, Gecer et al. (2019); Kanazawa et al. (2018a); Sanyal et al. (2019); Shang et al. (2020) reconstruct the 3D shape of human faces or bodies using 3DMM (Blanz & Vetter, 1999) or SMPL (Loper et al., 2015) as a prior knowledge.”
Learning an animatable detailed 3D face model from in-the-wild images (2020, methodology): “We compare DECA with publicly available methods, namely 3DDFA-V2 [Guo et al. 2020], Deng et al. [2019], RingNet [Sanyal et al. 2019], PRNet [Feng et al. 2018b], 3DMM-CNN [Tran et al. 2017] and Ex-treme3D [Tran et al. 2018].”
3D Morphable Face Models—Past, Present, and Future (2019, neutral): “By far the most commonly used features are landmarks (alternatively known as keypoints or fiducial points) which often used for initialization and are still important in much of the state-of-the-art, e.g., [Sanyal et al. 2019].”
I M Avatar: Implicit Morphable Head Avatars from Videos (2021, methodology): “Conventional methods [15, 17, 19, 20, 41, 43, 45, 47] based on morphable mesh models [3, 27, 36] can fit the shape and texture parameters to images for a given subject.”