Production-level facial performance capture using deep convolutional neural networks

By Samuli Laine, Tero Karras, Timo Aila, Antti Herva, Shunsuke Saito, Ronald Yu, Hao Li, Jaakko Lehtinen. 2016

104 citations recorded by the source corpus.

We present a real-time deep learning framework for video-based facial performance capture---the dense 3D tracking of an actor's face given a monocular video. Our pipeline begins with accurately capturing a subject using a high-end production facial capture pipeline based on multi-view stereo tracking and artist-enhanced animations. With 5--10 minutes of captured footage, we train a convolutional neural network to produce high-quality output, including self-occluded regions, from a monocular video sequence of that subject. Since this 3D facial performance capture is fully automated, our system can drastically reduce the amount of labor involved in the development of modern narrative-driven video games or films involving realistic digital doubles of actors and potentially hours of animated dialogue per character. We compare our results with several state-of-the-art monocular real-time facial capture techniques and demonstrate compelling animation inference in challenging areas such as eyes and lips.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Signal Processing

Citations in the corpus

Privacy in Deep Learning: A Survey (2020, background): “, has resulted in their deployment in numerous production systems [1, 2].”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, background): “While some learning based methods do pursue mesh prediction [15,37,19,25], they also rely on 3D supervision which is only available for restricted classes or in a synthetic setting.”
A Principled Approach to Learning Stochastic Representations for Privacy in Deep Neural Inference (2020, background): “The success of deep learning in many areas including vision, recommendation systems, natural language processing, etc., has heralded the adoption of Deep Neural Networks (DNNs) in production systems (Laine et al., 2017; Këpuska & Bohouta, 2018).”
Capture, Learning, and Synthesis of 3D Speaking Styles (2019, background): “[35] build subject-specific face-rigs from high-resolution face scans and animate these rigs with video-based animation systems.”
Multilinear Autoencoder for 3D Face Model Learning (2018, methodology): “[16], in which the linear 3DMM is initialized with principal component analysis, and refined during fine-tuning of the network.”
3D Morphable Face Models—Past, Present, and Future (2019, unknown): “For dense reconstruction tasks, some approaches [Laine et al. 2017] are trained based on images captured in a controlled multiview capture setup.”
SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild' (2017, background): “Recently many research works aim to reconstruct the shape of real world faces by learning from synthetic data; by fitting a 3DMM [35, 17, 34], by predicting a depth map and subsequent non-rigid deformation to obtain a mesh [28] and by regressing a normal map [36].”
InverseFaceNet: Deep Monocular Inverse Face Rendering (2017, methodology): “Recent techniques approach monocular face reconstruction by fitting active appearance models [1, 17], blendshape models [9, 18, 19, 61], affine face models [15, 16, 20, 46, 51, 54, 58, 62], mesh geometry [26, 33, 47, 48, 52], or volumetric geometry [24] to input images or videos.”