Face transfer with multilinear models

By Daniel Vlasic, Matthew Brand, Hanspeter Pfister, Jovan Popovic. 2005

553 citations recorded by the source corpus.

Face Transfer is a method for mapping videorecorded performances of one individual to facial animations of another. It extracts visemes (speech-related mouth articulations), expressions, and three-dimensional (3D) pose from monocular video or film footage. These parameters are then used to generate and drive a detailed 3D textured face mesh for a target identity, which can be seamlessly rendered back into target footage. The underlying face model automatically adjusts for how the target performs facial expressions and visemes. The performance data can be easily edited to change the visemes, expressions, pose, or even the identity of the target---the attributes are separably controllable. This supports a wide variety of video rewrite and puppetry applications.Face Transfer is based on a multilinear model of 3D face meshes that separably parameterizes the space of geometric variations due to different attributes (e.g., identity, expression, and viseme). Separability means that each of these attributes can be independently varied. A multilinear model can be estimated from a Cartesian product of examples (identities × expressions × visemes) with techniques from statistical analysis, but only after careful preprocessing of the geometric data set to secure one-to-one correspondence, to minimize cross-coupling artifacts, and to fill in any missing examples. Face Transfer offers new solutions to these problems and links the estimated model with a face-tracking algorithm to extract pose, expression, and viseme parameters.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Generating 3D faces using Convolutional Mesh Autoencoders (2018, background): “The existing state of the art 3D face representations mostly use linear transformations [41, 28, 42] or higher-order tensor generalizations [46, 12, 14].”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, background): “Moreover, they focus on transferring the expressions without the head movement [71, 75, 76, 77, 80].”
X2Face: A network for controlling face generation by using images, audio, and pose codes (2018, methodology): “Given a driving and source video sequence, a 3DMM or 3D mesh can be obtained and used to model both the driving and source face [10, 40, 43].”
Displaced dynamic expression regression for real-time facial tracking and animation (2014, methodology): “1996; DeCarlo and Metaxas 2000], and data-driven face models [Pighin et al. 1999; Blanz and Vetter 1999; Vlasic et al. 2005].”
Deep video portraits (2018, methodology): “…transferred via dense motion ields [Averbuch-Elor et al. 2017; Liu et al. 2001; Suwajanakorn et al. 2015], parameters [Thies et al. 2016, 2018; Vlasic et al. 2005], or by warping candidate frames that are selected based on the facial motion [Dale et al. 2011], appearance metrics…”
A 3D Morphable Model Learnt from 10,000 Faces (2016, methodology): “[30] modeled the combined effect of identity and expression variation on the facial shape by using a multilinear model.”
State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications (2018, methodology): “Blendshapes can be obtained from user-speciﬁc expressions [WBLP11,WBGB16], generated via statistical analysis from a large facial expression database [CWZ ∗ 14, VBPP05], or can be hand-crafted by animation artists [BWP13].”
Tensor Decompositions and Applications (2009, background): “Examples include signal processing [62, 196, 47, 43, 68, 80, 173, 60, 61], numerical linear algebra [87, 63, 64, 132, 244, 133, 149], computer vision [229, 230, 231, 190, 236, 237, 193, 102, 235], numerical analysis [22, 108, 23, 89, 114, 88, 115], data mining [190, 4, 157, 211, 5, 209, 210, 44, 14]”
Capture, Learning, and Synthesis of 3D Speaking Styles (2019, methodology): “Tensor-based multilinear face models [5, 11, 12, 19, 57, 63] and linear models [55] are widely used to build personalized face-rigs.”
Text-based editing of talking-head video (2019, background): “Facial video reenactment has been an active area of research [Averbuch-Elor et al. 2017; Garrido et al. 2014; Kemelmacher-Shlizerman et al. 2010; Li et al. 2014; Liu et al. 2001; Suwajanakorn et al. 2017; Vlasic et al. 2005].”
Controllable Face Privacy (2015, methodology): “Altering facial attributes would thus be more difficult with Tensorface.”
Learning a model of facial shape and expression from 4D scans (2017, methodology): “Consequently one could learn a mapping from our space to semantic attributes as shown in other works [Allen et al. 2003; Hasler et al. 2009; Vlasic et al. 2005] or train a localized space as proposed by Neumann et al.”
Review of statistical shape spaces for 3D data with comparative analysis for human faces (2012, methodology): “On the other hand, less detail and a lower-dimensional model are generally preferable for applications such as face and expression recognition [72, 73], model-based manipulation of images and videos [7, 12, 13, 14], and control of virtual avatars [74].”
Reconstruction of Personalized 3D Face Rigs from Monocular Video (2016, background): “Multilinear models represent both identity and expression variations, and can be learned from laser scan databases [Vlasic et al. 2005].”