Learning Detailed Face Reconstruction from a Single Image

By Elad Richardson, Matan Sela, Roy Or-El, Ron Kimmel. 2016

359 citations recorded by the source corpus.

Reconstructing the detailed geometric structure of a face from a given image is a key to many computer vision and graphics applications, such as motion capture and reenactment. The reconstruction task is challenging as human faces vary extensively when considering expressions, poses, textures, and intrinsic geometries. While many approaches tackle this complexity by using additional data to reconstruct the face of a single subject, extracting facial surface from a single image remains a difficult problem. As a result, single-image based methods can usually provide only a rough estimate of the facial geometry. In contrast, we propose to leverage the power of convolutional neural networks to produce a highly detailed face reconstruction from a single image. For this purpose, we introduce an end-to-end CNN framework which derives the shape in a coarse-to-fine fashion. The proposed architecture is composed of two main blocks, a network that recovers the coarse facial geometry (CoarseNet), followed by a CNN that refines the facial features of that geometry (FineNet). The proposed networks are connected by a novel layer which renders a depth image given a mesh in 3D. Unlike object recognition and detection problems, there are no suitable datasets for training CNNs to perform face geometry reconstruction. Therefore, our training regime begins with a supervised phase, based on synthetic images, followed by an unsupervised phase that uses only unconstrained facial images. The accuracy and robustness of the proposed model is demonstrated by both qualitative and quantitative evaluation tests.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Hybrid Model for Identity Obfuscation by Face Replacement (2018, methodology): “Some reconstruction methods go beyond the low-dimensional parametric models [24,6,16,17,21,19] to capture more face detail, but they lack parametric control of captured highfrequency details.”
Extreme 3D Face Reconstruction: Seeing Through Occlusions (2017, background): “The SfS method we used is rather slow, taking minutes to compute a single depth map.”
Modeling Facial Geometry Using Compositional VAEs (2018, background): “With the advent of Deep Learning, there have been several attempts at using deep nets for data-driven face reconstruction [35, 11, 29].”
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set (2019, methodology): “Some of them apply CNNs to regress 3DMM coefﬁcients [37, 13, 2, 49, 51, 16], some use multi-step schemes to add correction or details onto coarse 3DMM predictions [38, 48, 52, 18], while others advocate direct model-free reconstruction [45, 53, 46, 14].”
MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction (2017, background): “Unlike [41, 42], our network is trained completely unsupervised on real images, while they use a synthetic training corpus that lacks realistic features.”
StyleRig: Rigging StyleGAN for 3D Control Over Portrait Images (2020, methodology): “The computer vision and graphics communities have a rich history of modeling face rigs [21, 25, 26, 31].”
Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning (2019, background): “Recent advances in 3D face reconstruction [40, 42, 41, 43, 9], material inference [28, 7] and other 3D reconstruction tasks [51, 38, 34, 14, 23, 35, 39] have leveraged some other forms of differentiable rendering layers to obtain gradient flows in the neural networks.”
Learning to Infer Implicit Surfaces without 3D Supervision (2019, background): “Aside from silhouettes, shading and appearances in image space also provides supervision cues for learning fine-grained shape representations in category specific domains such as 3D face reconstruction [31, 32, 33, 34, 35] and material inference [36, 37, 38].”
Differentiable Monte Carlo ray tracing through edge sampling (2018, background): “Recently,it is increasingly popular for deep learning methods to incorporate a differentiable rendering layer in their architecture (e.g. [Liu et al. 2017; Richardson et al. 2017]).”
Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation (2017, methodology): “While a similar representation was used in [34, 48] as feedback channel for an iterative network, the facial recovery was still restricted to the span of a facial morphable model.”
Deep Face Recognition: A Survey (2018, background): “one to many generating many patches or images of the pose variability from a single image 3D model [109], [107], [129], [130], [48] [57], [155], [154] 2D deep model [231], [221], [141] data augmentation [99], [228], [46] [177], [145], [146], [150], [160]”
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision (2019, background): “ure representations. Using an analytic expression for the gradient of the depth @d^ @ wrt. the network parameters , we learn implicit 3D representations f from 2D images. to make it differentiable [4,11,15,21,33,36,43,44,47,50, 58,59,62,75,76,79,88]. While yielding compelling results, they are restr”
3D Morphable Face Models—Past, Present, and Future (2019, neutral): “s problem in the future would be better models that include all these components. To leverage the advantages of both real as well as synthetic training data, many current approaches [Kim et al. 2017; Richardson et al. 2017] are trained on a mixture of data from these two domains. The hope here is th”
Text-based editing of talking-head video (2019, methodology): “Besides model parameters, other approaches regress detailed depth maps [Richardson et al. 2017; Sela et al. 2017], or 3D displacements [Cao et al.”
Unsupervised Training for 3D Morphable Model Regression (2018, methodology): “Previous approaches have explored synthesizing training pairs of image and morphable model coordinates in a preprocess [20, 21, 29], or training an imageFigure 1.”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “Richardson et al. [128], [129] generate synthetic training data by drawing random samples from the morphable model and rendering the resulting faces.”