Deep video portraits

By Hyeongwoo Kim, Pablo Garrido, Ayush Kumar Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, Christian Theobalt. ACM Transactions on Graphics, 2018

780 citations recorded by the source corpus.

We present a novel approach that enables photo-realistic re-animation of portrait videos using only an input video. In contrast to existing approaches that are restricted to manipulations of facial expressions only, we are the first to transfer the full 3D head position, head rotation, face expression, eye gaze, and eye blinking from a source actor to a portrait video of a target actor. The core of our approach is a generative neural network with a novel space-time architecture. The network takes as input synthetic renderings of a parametric face model, based on which it predicts photo-realistic video frames for a given target actor. The realism in this rendering-to-video transfer is achieved by careful adversarial training, and as a result, we can create modified target videos that mimic the behavior of the synthetically-created input. In order to enable source-to-target video re-animation, we render a synthetic target video with the reconstructed head animation parameters from a source video, and feed it into the trained network - thus taking full control of the target. With the ability to freely recombine source and target parameters, we are able to demonstrate a large variety of video rewrite applications without explicitly modeling hair, body or background. For instance, we can reenact the full head using interactive user-controlled editing, and realize high-fidelity visual dubbing. To demonstrate the high quality of our output, we conduct an extensive series of experiments and evaluations, where for instance a user study shows that our video edits are hard to detect.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/efb0a4574736a976b6dc3c23be45e5fc0ac6fe41

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, background): “…[48], such as unconditional image synthesis [21, 23, 32, 33, 34, 49, 61], image translation [8, 12, 27, 29, 46, 47, 58, 67, 73, 84, 102, 103], text-to-image translation [62, 90, 95], image processing [17,18,28,36,39,41,42,44,72,78,89,94], and video synthesis [2,10,35,45,50,60,63,69,82,83,101].”
Disentangling in Latent Space by Harnessing a Pretrained Generator (2020, background): “By comparison, face reenactment methods transfer expression and head pose (sometimes also illumination and eye gaze) from a source video to a target identity [32, 56, 57, 4].”
HoloGAN: Unsupervised Learning of 3D Representations From Natural Images (2019, background): “…adversarial networks (GANs), have achieved impressive results in generating images of high resolution and visual quality [1, 5, 27, 28, 62], while their conditional versions have achieved great progress in image-to-image translation [23, 50], image editing [11, 12, 60] or motion transfer [6, 30].”
DeepVoxels: Learning Persistent 3D Feature Embeddings (2018, background): “Approaches that convert synthetic images into photorealistic imagery have been proposed for the special case of human bodies [64, 2] and faces [28].”
Protecting World Leaders Against Deep Fakes (2019, background): “The temporal consistency of these videos was improved by conditioning the network with multiple frames allowing the network to see in time[15].”
Neural Sparse Voxel Fields (2020, methodology): “Other methods (Thies et al., 2019; Kim et al., 2018; Liu et al., 2019a, 2020; Meshry et al., 2019; Martin Brualla et al., 2018; Aliev et al., 2019) first render a given geometry with explicit or neural textures into coarse RGB images or feature maps which are then translated into high-quality images”
Deepfake Video Detection through Optical Flow Based CNN (2019, methodology): “From one side, an effort has been spent to develop new kinds of effective synthesized video generation techniques such as Face2Face [14], Deep Video Portraits [7], StarGAN [5] and Deep Fake(1).”
Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos (2018, methodology): “There are several state-of-the-art methods for detecting images or videos generated by computer using, for example, a deepfake technique for face swapping [6], the Face2Face method for facial reenactment [1], or the deep video portraits technique [2] for the purpose of forgery.”
Cross-Domain and Disentangled Face Manipulation With 3D Guidance (2021, background): “differentiable graphics pipelines [38], [39], [40], combining the traditional pipelines with learnable components [41],”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “[125] proposed an approach that allows the full reanimation of portrait videos by an actor, such as changing head pose, eye gaze, and blinking, rather than just modifying the facial expression of the target identity and thus produced photorealistic dubbing results.”
Few-Shot Adversarial Learning of Realistic Neural Talking Head Models (2019, background): “These systems, therefore, require a several-minutes-long video [20, 37] or a large dataset of photographs [16] as well as hours of GPU training in order to create a new personalized talking head model.”
HyperNeRF (2021, methodology): “2019] or conventional renderings [Fried et al. 2019; Kim et al. 2018; Martin-Brualla et al. 2018; Meshry et al. 2019] as a harness with which neural rendering can then be guided.”
FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping (2019, methodology): “2, 3 [20] Hyeongwoo Kim, Pablo Carrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt.”
Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations (2019, background): “[20] extends these approaches by allowing to manipulate the 3-D head position, head rotation, face expression, eye gaze, and eye blinking using a generative neural network.”