Realistic Speech-Driven Facial Animation with GANs

By Konstantinos Vougioukas, Stavros Petridis, Maja Pantic. International Journal of Computer Vision, 2019

334 citations recorded by the source corpus.

Speech-driven facial animation is the process that automatically synthesizes talking characters based on speech signals. The majority of work in this domain creates a mapping from audio features to visual features. This approach often requires post-processing using computer graphics techniques to produce realistic albeit subject dependent results. We present an end-to-end system that generates videos of a talking head, using only a still image of a person and an audio clip containing speech, without relying on handcrafted intermediate features. Our method generates videos which have (a) lip movements that are in sync with the audio and (b) natural facial expressions such as blinks and eyebrow movements. Our temporal GAN uses 3 discriminators focused on achieving detailed frames, audio-visual synchronization, and realistic expressions. We quantify the contribution of each component in our model using an ablation study and we provide insights into the latent representation of the model. The generated videos are evaluated based on sharpness, reconstruction quality, lip-reading accuracy, synchronization as well as their ability to generate natural blinks.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/f06ab43069480c09d7170075a6ea74669a71f139

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset (2021, methodology): “While many works[7, 6, 18, 13, 51, 5, 48, 29, 42, 49, 4] make great efforts to synthesize realistic-looking videos, the generation of high-resolution videos is still a challenge.”
AI-generated characters for supporting personalized learning and well-being (2021, methodology): “The second approach utilizes an end-to-end speech-driven facial animation system based on a temporal GAN 65 .”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, background): “On the other hand, subject-agnostic models [4, 9, 11, 20, 22, 24, 30, 56, 60, 68, 70, 81, 82, 86, 92, 93, 100] only need a single image of the target person, who is not seen during training, to synthesize arbitrary motions.”
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild (2020, background): “lso have an additional overhead of requiring clean training data of each target speaker to generate for that speaker. Another limitation of existing works is in terms of the vocabulary. Several works [5, 26, 28] train on datasets with a limited set of words such as GRID [10] (56 words), TIMIT [14] a”
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis (2021, methodology): “Earlier methods try to synthesize the lip motions that fulfill the training data of a still facial image [5, 13, 8, 42, 7, 37].”
The Creation and Detection of Deepfakes (2020, background): “[164], the same authors improve the temporal coherence by splitting the video discriminator into two: (1) for temporal realism in mouth to audio synchronization and (2) for temporal realism in overall facial expressions.”
Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection (2020, background): “Face manipulation methods in particular have raised considerable concerns due to their ability to alter a person’s identity [38, 30, 34], expression [52, 51, 44], or lip movements [50, 57, 56] to match the face in a given target video.”
MEAD: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation (2020, background): “Konstantinos [52] uses three discriminators to enhance details, synchronization, and realistic expressions like blinks and eyebrow movements.”
What comprises a good talking-head video generation?: A Survey and Benchmark (2020, methodology): “4 shows the t-SNE plot of different ESD features on CREMA-D testing set, from where we observe that the group boundaries of ESD feature extracted from baseline method, Wiles et al. [18], and Zakharov et al. [20] are more clear than the ESD feature extracted from synthesized videos produced by Vougio”
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (2021, methodology): “We use SSIM [60] to account for the generation quality, and the cumulative probability blur detection ( CPBD ) measure [37] is adopted from [55] to evaluate the sharpness of the results.”
MakeItTalk: Speaker-Aware Talking Head Animation (2020, methodology): “We notice that the videos generated by Chen et al. [2019]; Vougioukas et al. [2019] predict primarily the lip region on cropped faces and therefore miss the head poses.”
Deepfake Video Detection Using Convolutional Vision Transformer (2021, background): “The authors in [58] presented a system that generates videos of talking heads with natural facial expressions such as eye blinking.”