Realtime performance-based facial animation

By Thibaut Weise, Sofien Bouaziz, Hao Li, Mark Pauly. ACM Transactions on Graphics, 2011

632 citations recorded by the source corpus.

This paper presents a system for performance-based character animation that enables any user to control the facial expressions of a digital avatar in realtime. The user is recorded in a natural environment using a non-intrusive, commercially available 3D sensor. The simplicity of this acquisition device comes at the cost of high noise levels in the acquired data. To effectively map low-quality 2D images and 3D depth maps to realistic facial expressions, we introduce a novel face tracking algorithm that combines geometry and texture registration with pre-recorded animation priors in a single optimization. Formulated as a maximum a posteriori estimation in a reduced parameter space, our method implicitly exploits temporal coherence to stabilize the tracking. We demonstrate that compelling 3D facial dynamics can be reconstructed in realtime without the use of face markers, intrusive lighting, or complex scanning hardware. This makes our system easy to deploy and facilitates a range of new applications, e.g. in digital gameplay or social interactions.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/88129ed70b949058308f293b38de0d5b0fd9b360

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

A deep learning approach for generalized speech animation (2017, neutral): “The main alternative to hand animation is performance-driven animation using facial motion capture of an actor’s face [Beeler et al. 2011; Cao et al. 2015, 2013; Fyffe et al. 2014; Huang et al. 2011; Li et al. 2013; Weise et al. 2011; Weng et al. 2014; Zhang et al. 2004].”
Review of statistical shape spaces for 3D data with comparative analysis for human faces (2012, methodology): “Weise et al. [52] proposed a method to track a three-dimensional face model over time using prior information on the deformation model, and to use the tracked model to drive the animation of a virtual character in real-time.”
State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications (2018, methodology): “These challenges make lip motion hard to capture with current state-of-the-art monocular face tracking approaches [WBLP11, LYYB13, CWS ∗ 13, BWP13, SWTC14, TZN ∗ 15, SSS14, TZS ∗ 16a, GZC ∗ 16, WSXC16], especially if the motion contains signiﬁcant amounts of stretching, bending and rolling.”
Online modeling for realtime facial animation (2013, methodology): “Several realtime methods for face tracking have been proposed that require only a single video camera [Chai et al. 2003; Amberg et al. 2009; Saragih et al. 2011] or consumer-level RGB-D camera, such as the Microsoft Kinect [Weise et al. 2011; Baltrušaitis et al. 2012].”
FaceScape: A Large-Scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction (2020, methodology): “M 0 is given by M 0 = max(0 , 1 There are many existing performance driven facial animation methods generating blendshape weights with depth camera[54, 29, 9] or single RGB camera[11, 10, 14].”
Deep video portraits (2018, background): “Performance-driven 3D head animation methods [Cao et al. 2015, 2014a, 2016; Hu et al. 2017; Ichim et al. 2015; Li et al. 2015; Olszewski et al. 2016; Weise et al. 2011] are related to our work, but have orthogonal methodology and application goals.”
Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation (2017, methodology): “This feature map is a deformation agnostic representation, which is useful for applications such as facial motion capture [44], face normalization [49] and texture mapping [50].”
Capture, Learning, and Synthesis of 3D Speaking Styles (2019, methodology): “Li et al. [36] and Weise et al. [60] capture example-based rigs in an ofﬂine calibration procedure to build personalized face-rigs, Bouaziz et al. [6] use a generic identity model.”
3D Face Pose Tracking using Low Quality Depth Cameras (2013, methodology): “3D face pose tracking is becoming an important task for many research domains in computer vision like Human-Computer Interaction and face analysis (Weise et al., 2011; Cai et al., 2010; Maurel et al., 2008) and recognition (Kim et al., 2008).”
Face2Face: Real-Time Face Capture and Reenactment of RGB Videos (2016, background): “Impressive results have been achieved, both based on RGB [8, 6] as well as RGB-D data [31, 10, 21, 4, 16].”
Production-level facial performance capture using deep convolutional neural networks (2016, methodology): “A lot of work has also been done for RGB-D sensors, such as Microsoft Kinect Fusion, e.g., [Bouaziz et al. 2013; Hsieh et al. 2015b; Li et al. 2013; Thies et al. 2015; Weise et al. 2011].”
Registration of 3D Point Clouds and Meshes: A Survey from Rigid to Nonrigid (2013, background): “We ﬁrst point out the connection, leading to the discussion of three core registration components in the light of the overﬁtting problem: model selection , correspondences & constraints and optimization .”