Photorealistic Facial Texture Inference Using Deep Neural Networks

By Shunsuke Saito, Lingyu Wei, Liwen Hu, Koki Nagano, Hao Li. 2016

138 citations recorded by the source corpus.

We present a data-driven inference method that can synthesize a photorealistic texture map of a complete 3D face model given a partial 2D view of a person in the wild. After an initial estimation of shape and low-frequency albedo, we compute a high-frequency partial texture map, without the shading component, of the visible face area. To extract the fine appearance details from this incomplete input, we introduce a multi-scale detail analysis technique based on mid-layer feature correlations extracted from a deep convolutional neural network. We demonstrate that fitting a convex combination of feature correlations from a high-resolution face database can yield a semantically plausible facial detail description of the entire face. A complete and photorealistic texture map can then be synthesized by iteratively optimizing for the reconstructed feature correlations. Using these high-resolution textures and a commercial rendering framework, we can produce high-fidelity 3D renderings that are visually comparable to those obtained with state-of-the-art multi-view face capture systems. We demonstrate successful face reconstructions from a wide range of low resolution input images, including those of historical figures. In addition to extensive evaluations, we validate the realism of our results using a crowdsourced user study.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow (2018, background): “2016], to generate photorealistic face textures [Saito et al. 2016], and to robustly transfer image styles between photographs [Luan et al.”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, background): “An exception is texture inference on human faces [2,22,23,28], but these approaches require a large-set of 3D ground truth data with high quality texture maps.”
SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild' (2017, methodology): “proposes a method to synthesize a photorealistic albedo from a partial albedo obtained by traditional methods [27].”
Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation (2017, background): “• A three-dimensional morphable model - a prior lowdimensional linear subspace of plausible facial geometries which allows an efficient, yet rough, recovery of a facial structure [4, 6, 49, 36, 23, 33, 43],”
GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction (2019, background): “In these papers [32, 41], the texture looks excellent when rendered under professional renderers (e.”
3D Morphable Face Models—Past, Present, and Future (2019, unknown): “Moreover, a range of generative methods that synthesize facial textures have been proposed, e.g., by Saito et al. [2017], Slossberg et al. [2018], Deng et al. [2018], Lombardi et al. [2018], Nagano et al. [2018] and Yamaguchi et al. [2018].”
paGAN (2018, unknown): “…dynamic face models are suitable for single-view face modeling and tracking applications [Blanz and Vetter 1999; Bouaziz et al. 2013; Cao et al. 2014, 2016; Garrido et al. 2016; Hsieh et al. 2015; Hu et al. 2017; Li et al. 2013; Saito et al. 2016, 2017; Thies et al. 2015, 2016a; Weise et al. 2011].”
Learning an animatable detailed 3D face model from in-the-wild images (2020, background): “Instead of capturing high-frequency geometric details, somemethods reconstruct coarse facial geometry along with high-fidelity textures [Gecer et al. 2019; Saito et al. 2017; Slossberg et al. 2018; Yamaguchi et al. 2018].”
X2Face: A network for controlling face generation by using images, audio, and pose codes (2018, background): “fitting of 3DMMs by including high level details [34, 41], taking into account additional images [33] or 3D scans [4], or by learning 3DMM parameters directly from RGB data without ground truth labels [2, 39].”
Texture Fields: Learning Texture Representations in Function Space (2019, methodology): “However, current approaches based on mesh representations usually assume known topology and a predefined template mesh which limits these approaches to specific object categories such as birds [18] or faces [8,36].”