X2Face: A network for controlling face generation by using images, audio, and pose codes

By Olivia Wiles, A. Sophia Koepke, Andrew Zisserman. European Conference on Computer Vision, 2018

461 citations recorded by the source corpus.

The objective of this paper is a neural network model that controls the pose and expression of a given face, using another face or modality (e.g. audio). This model can then be used for lightweight, sophisticated video and image editing.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9ea992f009492888c482d5f4006281eaa8b758e7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

MAST: A Memory-Augmented Self-Supervised Tracker (2020, background): “Due to the abundance of online data [1, 4, 11, 14, 15, 22, 24, 25, 26, 27, 32, 38, 43, 59, 63, 67, 68], various ideas have been explored to learn representations by exploiting the spatio-temporal information in videos.”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “X2face [34] 2Encoder2Decoder § Facial landmark features § 256-D audio features § VGG Face dataset § VoxCeleb dataset § AFLW dataset 256×256 § Wrinkle artifacts § Lack of gaze adaption”
Animating Arbitrary Objects via Deep Motion Transfer (2018, methodology): “As proposed in [46], we first evaluate Monkey-Net on the task of video reconstruction.”
Deep Audio-visual Learning: A Survey (2020, background): “[102] put forward a self-supervising framework called X2Face to learn the embedded features and generate target facial motions.”
Generative Adversarial Networks for Face Generation: A Survey (2022, background): “Dynamic face generation has become the focus of facial GANs for its importance in real-world applications [10, 130, 141].”
FaceFormer: Speech-Driven 3D Facial Animation with Transformers (2021, unknown): “photorealistic talking-head animation [12, 15, 42, 52, 63, 67, 69].”
The Creation and Detection of Deepfakes (2020, background): “In [172] the authors propose X2Face which can reenact xt with xs or some other modality such as audio or a pose vector.”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, background): “Over the past few years rapid progress has been made in the direction of disentangling the latent space of deep models into dimensions that account for generic factors of variation, such as identity and low-dimensional transformations [8, 52, 28, 51, 39], or even non-rigid, dense deformations from a”
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis (2021, methodology): “Earlier methods try to synthesize the lip motions that fulfill the training data of a still facial image [5, 13, 8, 42, 7, 37].”
Motion Representations for Articulated Animation (2021, methodology): “For quantitative metrics, prior works [39, 28, 27] use video reconstruction accuracy as a proxy for image animation quality.”
Few-Shot Adversarial Learning of Realistic Neural Talking Head Models (2019, methodology): “X2Face, as a feed-forward method, is simply initialized via the training frames, while Pix2pixHD and our model are being additionally fine-tuned for 40 epochs on the few-shot set.”
Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss (2019, methodology): “All of them are trained on LRW dataset while Chung et al. [3] require extra VGG-M network pretrained on VGG Face dataset [25] and Wilels et al. [35] need extra MFCC feature extractor pretrained by [5].”
Detecting Photoshopped Faces by Scripting Photoshop (2019, methodology): “To generate out-of-domain warps, we randomly sampled the latent space of the optical flow generator in the X2face model [6] to generate warps.”
Self-supervised learning of a facial attribute embedding from video (2018, background): “Our approach builds in particular on those that use frame synthesis [8, 11, 22, 44, 56, 58], though for us synthesis is a proxy task rather than the end goal.”
PIE (2020, methodology): “We also compare to X2Face [Wiles et al. 2018], which samples a learned embedded face in order to synthesize portrait images with different poses and expressions.”
Cross-Domain and Disentangled Face Manipulation With 3D Guidance (2021, background): “[20] propose to generate a dense motion field by using”
Text-based editing of talking-head video (2019, methodology): “ng-based image-to-image translation [Isola et al. 2017]. Some recent approaches enable the synthesis of controllable facial animations from single images [Averbuch-Elor et al. 2017; Geng et al. 2018; Wiles et al. 2018]. Nagano et al.[2018] recently showed how to estimate a controllable avatar of a p”
First Order Motion Model for Image Animation (2020, methodology): “It’s important to note that compared to X2Face [40], we obtain more natural videos where faces move freely within the bounding box.”