Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression

By Aaron S. Jackson, Adrian Bulat, Vasileios Argyriou, Georgios Tzimiropoulos. 2017

471 citations recorded by the source corpus.

3D face reconstruction is a fundamental Computer Vision problem of extraordinary difficulty. Current systems often assume the availability of multiple facial images (sometimes from the same subject) as input, and must address a number of methodological challenges such as establishing dense correspondences across large facial poses, expressions, and non-uniform illumination. In general these methods require complex and inefficient pipelines for model building and fitting. In this work, we propose to address many of these limitations by training a Convolutional Neural Network (CNN) on an appropriate dataset consisting of 2D images and 3D facial models or scans. Our CNN works with just a single 2D facial image, does not require accurate alignment nor establishes dense correspondence between images, works for arbitrary facial poses and expressions, and can be used to reconstruct the whole 3D facial geometry (including the non-visible parts of the face) bypassing the construction (during training) and fitting (during testing) of a 3D Morphable Model. We achieve this via a simple CNN architecture that performs direct regression of a volumetric representation of the 3D facial geometry from a single 2D image. We also demonstrate how the related task of facial landmark localization can be incorporated into the proposed framework and help improve reconstruction quality, especially for the cases of large poses and facial expressions. Code and models will be made available at http://aaronsplace.co.uk

Subfields: Computer Vision and Pattern Recognition, Neurology, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Neurology

In Signal Processing

Citations in the corpus

Generating 3D faces using Convolutional Mesh Autoencoders (2018, methodology): “[25] use a volumetric face representation in their CNN-based framework.”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, background): “Jackson et al. [132], for example, pro-pose a Volumetric Regression Network (VRN), which takes as input 2D images and predicts their corresponding 3D binary volume instead of a 3DMM. Unlike [127], the approach can deal with a wide range of expressions, poses and occlusions without alignment and…”
Extreme 3D Face Reconstruction: Looking Past Occlusions (2017, methodology): “Deep methods were also recently proposed for this purpose [7, 11, 23, 45, 46].”
Shape-shifting structured lattices via multimaterial 4D printing (2019, methodology): “5A) generated from a painting of Gauss through a machine learning algorithm (23) (SI Appendix).”
Learning an animatable detailed 3D face model from in-the-wild images (2020, methodology): “However, all these methods require explicit 3D supervision, which is provided either by an optimization-based model fitting [Feng et al. 2018b; Güler et al. 2017; Jackson et al. 2017; Wei et al. 2019] or by synthetic data generated by sampling a statistical face model [Dou et al.”
Learning to Regress 3D Face Shape and Expression From an Image Without 3D Supervision (2019, methodology): “Learning with 3D supervision: Deep learning methods are quickly replacing the optimization-based approaches [35, 39, 19, 16].”
DeepHuman: 3D Human Reconstruction From a Single Image (2019, methodology): “We use extended Binary Cross-Entropy (BCE) loss for the reconstruction of occupancy volume [22]:”
InverseFaceNet: Deep Monocular Inverse Face Rendering (2017, methodology): “Recent techniques approach monocular face reconstruction by fitting active appearance models [1, 17], blendshape models [9, 18, 19, 61], affine face models [15, 16, 20, 46, 51, 54, 58, 62], mesh geometry [26, 33, 47, 48, 52], or volumetric geometry [24] to input images or videos.”
Towards Fast, Accurate and Stable 3D Dense Face Alignment (2020, methodology): “network [28,18], achieving the state-of-the-art performance.”
Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction (2020, methodology): “We use extended cross-entropy losses [9] for training.”
Unsupervised Training for 3D Morphable Model Regression (2018, background): “Other recent deep learning approaches predict depth maps [25] or voxel grids [10], trading off a compact and interpretable output mesh for more faithful reproductions of the input image.”
Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network (2018, methodology): “[18] develops a volumetric representation of 3D face and uses a network to regress it from a 2D image.”
BodyNet: Volumetric Inference of 3D Human Body Shapes (2018, unknown): “Our network minimizes the binary cross-entropy loss after applying the sigmoid function on the network output similar to [46]:”
Artificial intelligence in the creative industries: a review (2020, methodology): “(b) 2D to 3D face conversion generated using the algorithm proposed in [201]”
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set (2019, background): “Also note that PRN has We further qualitatively compare with several learning-based methods including VRN [25], 3DDFA [57], and Liu et al .”
Convolutional Mesh Regression for Single-Image Human Shape Reconstruction (2019, background): “[12] also propose a form of volumetric reconstruction by extending their recent face reconstruction network [11] to work for full body images.”
Deep learning for cellular image analysis (2019, unknown): “Augmented microscopy may be extended to predict 3D images from 2D data 153 or to obtain high-quality confocal images from wide-field microscopy, and object tracking could be adapted to automate cell tracking and lineage construction.”
EMOCA: Emotion Driven Monocular Face Capture and Animation (2022, methodology): “Model-free approaches directly regress 3D meshes [18,20,28,33,40,70,74,79,93,95,97] or voxels [38] from an image, or optimize a Signed Distance Function (SDF) [61] to fit a face image.”
Extreme 3D Face Reconstruction: Seeing Through Occlusions (2017, methodology): “Deep methods were also recently proposed for this purpose [6, 11, 24, 49, 51].”