Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation

By Matan Sela, Elad Richardson, Ron Kimmel. 2017

274 citations recorded by the source corpus.

It has been recently shown that neural networks can recover the geometric structure of a face from a single given image. A common denominator of most existing face geometry reconstruction methods is the restriction of the solution space to some low-dimensional subspace. While such a model significantly simplifies the reconstruction problem, it is inherently limited in its expressiveness. As an alternative, we propose an Image-to-Image translation network that jointly maps the input image to a depth image and a facial correspondence map. This explicit pixel-based mapping can then be utilized to provide high quality reconstructions of diverse faces under extreme expressions, using a purely geometric refinement process. In the spirit of recent approaches, the network is trained only with synthetic data, and is then evaluated on “in-the-wild” facial images. Both qualitative and quantitative analyses demonstrate the accuracy and the robustness of our approach.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Hybrid Model for Identity Obfuscation by Face Replacement (2018, methodology): “he-wild images [12,13]. Such parametric models can act as strong regularizers for 3D face reconstruction problems, and have been widely used in optimization based [5,14,12,15,16,17] and learningbased [18,19,20,21,22,23] settings. Recently, a model-based face autoencoder (MoFA) has been introduced [4”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, background): “An exception is texture inference on human faces [2,22,23,28], but these approaches require a large-set of 3D ground truth data with high quality texture maps.”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “[133] use an Image-to-Image Translation Network based on U-Net [48] to estimate a depth image and a facial correspondence map.”
Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network (2018, background): “…contains the information of the whole face, which makes it diﬀerent from other 2D representations such as Projected Normalized Coordinate Code (PNCC)[11,14], an ordinary depth image[17] or quantized UV coordinates[28], which only re-serve the information of visible face region in the input image.”
Unsupervised Training for 3D Morphable Model Regression (2018, background): “[25] on 7 images from an 84-image test set developed”
InverseFaceNet: Deep Monocular Inverse Face Rendering (2017, methodology): “Recent techniques approach monocular face reconstruction by fitting active appearance models [1, 17], blendshape models [9, 18, 19, 61], affine face models [15, 16, 20, 46, 51, 54, 58, 62], mesh geometry [26, 33, 47, 48, 52], or volumetric geometry [24] to input images or videos.”
StyleRig: Rigging StyleGAN for 3D Control Over Portrait Images (2020, methodology): “parameters of a 3DMM by training it on large scale synthetic data [20, 25, 28].”
Tex2Shape: Detailed Full Human Body Geometry From a Single Image (2019, methodology): “, in analysis-by-synthesis fitting [43] or refinement, or in a trained neural network [44, 17].”
Multilinear Autoencoder for 3D Face Model Learning (2018, background): “[23] propose a model-free network that directly regresses from an input image to a depth image and a correspondence map, thereby allowing to recover fine-scale geometric detail.”
Learning to Regress 3D Face Shape and Expression From an Image Without 3D Supervision (2019, methodology): “[27] use a synthetic dataset to generate an image-to-depth mapping and a pixel-to-vertex mapping, which are combined to generate the face mesh.”
EMOCA: Emotion Driven Monocular Face Capture and Animation (2022, methodology): “1, 2, 5 [76] Matan Sela, Elad Richardson, and Ron Kimmel.”
Fake it till you make it: face analysis in the wild using synthetic data alone (2021, methodology): “[58] used 3DMMs to render training data for reconstructing detailed facial geometry.”
SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild' (2017, methodology): “We also compare against Pix2Vertex [28], which only estimate high resolution meshes.”
FaceScape: A Large-Scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction (2020, background): “[40] both proposed to firstly predict a rough facial shape and render it to the depth map, then refine the depth map to enhance the details from the registered source image.”
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set (2019, methodology): “Many methods resort to synthetic data or using 3D shapes ﬁtted by traditional methods as surrogate labels [37, 57, 45, 31, 14, 18].”
Extreme 3D Face Reconstruction: Looking Past Occlusions (2017, methodology): “The loose regularization required by [39] to produce ﬁne details results in a corrupt shape when occlusions are introduced.”
Alleviating Semantics Distortion in Unsupervised Low-Level Image-to-Image Translation via Structure Consistency Constraint (2022, methodology): “It has been applied to various vision tasks [13, 46, 49, 59, 65].”
Self-Supervised Multi-level Face Model Learning for Monocular Reconstruction at Over 250 Hz (2017, background): “[54] predicted a per-pixel depth map to deform and fill holes of a limited geometry subspace learned during training.”
GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction (2019, methodology): “The first such methods either trained regression DCNNs from image to the parameters of a 3DMM [39] or used a 3DMM to synthesize images [28, 18] and formulate an image-to-image translation problem using DCNNs to estimate the depth3 [34].”
Text-based editing of talking-head video (2019, methodology): “Besides model parameters, other approaches regress detailed depth maps [Richardson et al. 2017; Sela et al. 2017], or 3D displacements [Cao et al.”