Deep appearance models for face rendering

By Stephen Lombardi, Jason M. Saragih, Tomas Simon, Yaser Sheikh. ACM Transactions on Graphics, 2018

325 citations recorded by the source corpus.

We introduce a deep appearance model for rendering the human face. Inspired by Active Appearance Models, we develop a data-driven rendering pipeline that learns a joint representation of facial geometry and appearance from a multiview capture setup. Vertex positions and view-specific textures are modeled using a deep variational autoencoder that captures complex nonlinear effects while producing a smooth and compact latent representation. View-specific texture enables the modeling of view-dependent effects such as specularity. In addition, it can also correct for imperfect geometry stemming from biased or low resolution estimates. This is a significant departure from the traditional graphics pipeline, which requires highly accurate geometry as well as all elements of the shading model to achieve realism through physically-inspired light transport. Acquiring such a high level of accuracy is difficult in practice, especially for complex and intricate parts of the face, such as eyelashes and the oral cavity. These are handled naturally by our approach, which does not rely on precise estimates of geometry. Instead, the shading model accommodates deficiencies in geometry though the flexibility afforded by the neural network employed. At inference time, we condition the decoding network on the viewpoint of the camera in order to generate the appropriate texture for rendering. The resulting system can be implemented simply using existing rendering engines through dynamic textures with flat lighting. This representation, together with a novel unsupervised technique for mapping images to facial states, results in a system that is naturally suited to real-time interactive settings such as Virtual Reality (VR).

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6c1f0948aa1638a24cb01ec35ca3712242a3e3b2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis (2022, background): “One example is social telepresence in AR/VR, which aims to enable realistic face-to-face conversations between people in a virtual setting [34, 62].”
Neural Voice Puppetry: Audio-driven Facial Reenactment (2019, background): “It enables a variety of applications, ranging from consumer-level teleconferencing through photo-realistic virtual avatars [23,30,31] to movie production applications such as video dubbing [12,19].”
Neural volumes (2019, methodology): “In Lombardi et al. [2018], this technique was used to great effect in modeling the human face, where it was demonstrated that a neural network can be trained to compensate for geometric reconstruction and tracking inaccuracies through a view-dependent texture.”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, background): “Recent neural scene representations learn low-dimensional nonlinear representations of meshes [31, 66], point clouds [2, 41, 71], sphere sets [27], and dense volumetric grids [32, 58].”
Few-Shot Adversarial Learning of Realistic Neural Talking Head Models (2019, background): “A huge body of works is devoted to statistical modeling of the appearance of human faces [6], with remarkably good results obtained both with classical techniques [35] and, more recently, with deep learning [22, 25] (to name just a few).”
RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion (2022, background): “On the other hand, the available digital avatar data is very scarce as each avatar must be painstakingly created by a specialized 3D artist using a sophisticated creation pipeline [20, 35], especially for modeling hair and facial hair.”
PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization (2020, background): “While metrically accurate and precise reconstructions of humans is now possible with multi-view systems [12, 26], it has remained largely inaccessible to the general community due to its reliance on professional capture systems with strict environmental constraints (e.g., high number of cameras,…”
3D Morphable Face Models—Past, Present, and Future (2019, unknown): “Moreover, a range of generative methods that synthesize facial textures have been proposed, e.g., by Saito et al. [2017], Slossberg et al. [2018], Deng et al. [2018], Lombardi et al. [2018], Nagano et al. [2018] and Yamaguchi et al. [2018].”
FastNeRF: High-Fidelity Neural Rendering at 200FPS (2021, background): “Recently, neural rendering [10, 27, 17, 25, 19] has offered a disruptive alternative: involve a neural network in the rendering pipeline to output either images directly [24, 18, 27] or to model implicit functions that represent a scene appropriately [5, 23, 29, 25, 40].”
Mixture of volumetric primitives for efficient neural rendering (2021, methodology): “To handle the different radiometric properties of the cameras, e.g., color response and white balance, we employ per-camera color calibration based on 6 parameters (gain and bias per color channel) similar to [Lombardi et al. 2018], but pre-trained for all cameras once for each dataset.”
GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction (2019, methodology): “Although rendering networks (i.e. trained by VAE [25]) generates outstanding quality textures, each network is capable of storing up to few individuals whom should be placed in a 2Please see the supplementary materials for a comparison with [32, 41].”