Face2Face: Real-Time Face Capture and Reenactment of RGB Videos

By Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner. 2016

2,282 citations recorded by the source corpus.

We present a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion. To this end, we first address the under-constrained problem of facial identity recovery from monocular video by non-rigid model-based bundling. At run time, we track facial expressions of both source and target video using a dense photometric consistency measure. Reenactment is then achieved by fast and efficient deformation transfer between source and target. The mouth interior that best matches the re-targeted expression is retrieved from the target sequence and warped to produce an accurate fit. Finally, we convincingly re-render the synthesized target face on top of the corresponding video stream such that it seamlessly blends with the real-world illumination. We demonstrate our method in a live setup, where Youtube videos are reenacted in real time.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Hybrid Model for Identity Obfuscation by Face Replacement (2018, methodology): “ parametric face models [4] give us complete control of facial attributes and have demonstrated compelling results for applications such as face reconstruction, expression transfer and visual dubbing [4,5,6]. Importantly, using a parametric face model allows to control the identify of a person as we”
Generating 3D faces using Convolutional Mesh Autoencoders (2018, methodology): “A similar model with an additional albedo model was used within the Face2Face framework [43].”
Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning (2019, background): “It is becoming easier to manipulate existing videos of people: replacing a person’s facial expressions or movements with those of another person [6, 7], and/or overlaying synthesized speech which imitates a person’s voice [8].”
Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction (2020, methodology): “To estimate the per-frame expressions δi, we use an optimization-based facial reconstruction and tracking pipeline [41].”
FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping (2019, background): “It is extended for RGB references in Face2Face [39].”
GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction (2019, methodology): “A standard linear 3DMM fitting strategy [38] is used for face reconstruction followed by a number of steps for texture completion and refinement.”
First Order Motion Model for Image Animation (2020, methodology): “3D model) [4] and resorting to computer graphics techniques [6, 33].”
Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection (2021, methodology): “FF++ is a large-scale video dataset consisting of 1000 original videos that have been manipulated by four face manipulation methods: DeepFakes [1], Face2Face [40], FaceSwap[3], and Neural-Textures [39].”
Capture, Learning, and Synthesis of 3D Speaking Styles (2019, methodology): “Cao et al. [11, 12] use a regression-based face tracker to animate the face-rig and digital avatars, while Thies et al. [55] use a landmark-based face tracker and deformation transfer [50] to reenact monocular videos.”
FSGAN: Subject Agnostic Face Swapping and Reenactment (2019, methodology): “Suppressing the publication of such methods would not stop their development, but rather make them available to select few and potentially blindside [44] on FaceForen-sics++ [39].”
Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations (2019, methodology): “with different identities at random, where each parameter is sampled from a normal distribution with mean 0 and standard deviation of 0:7. For expressions, we use the blendshape model of Thies et al. [8] , and sample expression parameters uniformly in ( 0:4;1:6) . DeepVoxels dataset We use the datas”
Production-level facial performance capture using deep convolutional neural networks (2016, methodology): “accuracy. These methods are also usually only able to infer lowdimensional facial expressions—typically only a few blendshapes— reliably. More problems also arise in appearance based methods such as [Thies et al. 2016]. For example, relying on pixel constraints makes it possible only to track visibl”
Unsupervised Training for 3D Morphable Model Regression (2018, background): “Subsequent work ([4, 23, 29, 7, 14] and others) has applied a range of techniques to improve the accuracy and stability of the fitting, producing very accurate results under good conditions.”
A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentials (2022, methodology): “This set contains deepfakes generated by 8 non-GAN models , including Generative Flow (Glow) [37], Cascaded Refinement Networks (CRN) [11], Implicit Maximum Likelihood Estimation (IMLE) [43], second-order attention network (SAN) [15] and 4 other deepfake models (Deepfake [2], Face2Face [73], Faceswa”
SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild' (2017, background): “Recently many research works aim to reconstruct the shape of real world faces by learning from synthetic data; by fitting a 3DMM [35, 17, 34], by predicting a depth map and subsequent non-rigid deformation to obtain a mesh [28] and by regressing a normal map [36].”
X2Face: A network for controlling face generation by using images, audio, and pose codes (2018, methodology): “Given a driving and source video sequence, a 3DMM or 3D mesh can be obtained and used to model both the driving and source face [10, 40, 43].”
Exposing Deep Fakes Using Inconsistent Head Poses (2018, background): “Thanks to the recent developments of machine learning, the technologies for manipulating and fabricating images and videos have reached a new level of sophistication [1, 2, 3, 4, 5, 6, 7].”
FaceForensics++: Learning to Detect Manipulated Facial Images (2019, methodology): “More details of the reenactment process can be found in the original paper [56].”