High-quality passive facial performance capture using anchor frames

By Thabo Beeler, Fabian Hahn, Derek Bradley, Bernd Bickel, Paul A. Beardsley, Craig Gotsman, Robert W. Sumner, Markus H. Gross. ACM Transactions on Graphics, 2011

370 citations recorded by the source corpus.

We present a new technique for passive and markerless facial performance capture based on anchor frames. Our method starts with high resolution per-frame geometry acquisition using state-of-the-art stereo reconstruction, and proceeds to establish a single triangle mesh that is propagated through the entire performance. Leveraging the fact that facial performances often contain repetitive subsequences, we identify anchor frames as those which contain similar facial expressions to a manually chosen reference expression. Anchor frames are automatically computed over one or even multiple performances. We introduce a robust image-space tracking method that computes pixel matches directly from the reference frame to all anchor frames, and thereby to the remaining frames in the sequence via sequential matching. This allows us to propagate one reconstructed frame to an entire sequence in parallel, in contrast to previous sequential methods. Our anchored reconstruction approach also limits tracker drift and robustly handles occlusions and motion blur. The parallel tracking and mesh propagation offer low computation times. Our technique will even automatically match anchor frames across different sequences captured on different occasions, propagating a single mesh to all performances.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e48437eddcacfbeff01c3edd409369c9637a3357

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Production-level facial performance capture using deep convolutional neural networks (2016, background): “…high-quality facial capture methods used in a production setting still require complicated hardware and camera setups [Alexander et al. 2009; Beeler et al. 2011; Bhat et al. 2013; Borshukov et al. 2005; Fyffe et al. 2014; Vlasic et al. 2005; Weise et al. 2009a; Zhang et al. 2004] and a…”
Dynamic FAUST: Registering Human Bodies in Motion (2017, methodology): “Texture has been used for aligning isolated body parts such as faces and hands [6, 5, 17, 23, 43].”
Fusion4D (2016, methodology): “This is conceptually similar to the concept of a keyframe or anchor frame used in nonrigid tracking [Guo et al. 2015; Collet et al. 2015; Beeler et al. 2011], but this concept is extended for online nonrigid volumetric reconstruction.”
Total Moving Face Reconstruction (2014, methodology): “stereo approaches on input coming from multiple high resolution synchronized cameras which does not require markers but assumes calibration of the cameras and controlled lighting [7,8,11] or uncontrolled lighting[42].”
Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies (2018, methodology): “There exist multiview based methods showing excellent performance on high-quality facial scanning [5, 24] and facial motion capture [6, 9, 51].”
Online modeling for realtime facial animation (2013, background): “…3D acquisition, such as structured light scanners [Zhang et al. 2004; Weise et al. 2009] or multi-view camera systems [Furukawa and Ponce 2009; Bradley et al. 2010; Beeler et al. 2011; Valgaerts et al. 2012], have been developed more recently and proven efficient to capture finescale dynamics.”
Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction (2020, background): “While the explicit reconstruction of high quality geometry of the skin surface in a multi-view studio setup is tractable [4, 16, 52], hair is often approximated by retrieval and refinement of hair styles [18, 50], which leads to an unrealistic visual reproduction.”
Realtime facial animation with on-the-fly correctives (2013, methodology): “Correspondences are computed using variants of non-rigid registration and texture tracking [Zhang et al. 2004; Li et al. 2009; Furukawa and Ponce 2009; Bradley et al. 2010; Beeler et al. 2011] or more recently with a joint optimization between capture and tracking [Valgaerts et al.”
Adaptive 3D Face Reconstruction from Unconstrained Photo Collections (2017, background): “Synchronized multi-camera stereo setups [8], [20] can cap-”
Constraining dense hand surface tracking with elasticity (2020, methodology): “Dense correspondence techniques have made great recent progress in tracking faces [Beeler et al. 2011; Fyffe et al. 2015; Wu et al. 2018; Fyffe et al. 2017], often targeted at driving visual effects for films [Beeler et al.”
Displaced dynamic expression regression for real-time facial tracking and animation (2014, background): “In film and game production, special equipment, such as facial markers [Huang et al. 2011], camera arrays [Bradley et al. 2010; Beeler et al. 2011], and structured light projectors [Zhang et al. 2004; Weise et al. 2009], can be used to get 3D facial geometry of high fidelity.”
Modular primitives for high-performance differentiable rendering (2020, methodology): “The passive capture method of Beeler et al. [2011] first reconstructs each frame using a single-shot method [Beeler et al. 2010] and then builds frame-to-frame correspondences iteratively.”