Talking Face Generation by Adversarially Disentangled Audio-Visual Representation

By Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, Xiaogang Wang. AAAI Conference on Artificial Intelligence, 2018

490 citations recorded by the source corpus.

Talking face generation aims to synthesize a sequence of face images that correspond to a clip of speech. This is a challenging task because face appearance variation and semantics of speech are coupled together in the subtle movements of the talking face regions. Existing works either construct specific face appearance model on specific subjects or model the transformation between lip motion and speech. In this work, we integrate both aspects and enable arbitrary-subject talking face generation by learning disentangled audio-visual representation. We find that the talking face sequence is actually a composition of both subject-related information and speech-related information. These two spaces are then explicitly disentangled through a novel associative-and-adversarial training process. This disentangled representation has an advantage where both audio and video can serve as inputs for generation. Extensive experiments show that the proposed approach generates realistic talking face sequences on arbitrary subjects with much clearer lip motion patterns than previous work. We also demonstrate the learned audio-visual representation is extremely useful for the tasks of automatic lip reading and audio-video retrieval.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/1816f98e2a4dd54690c2689cf529699d8843e847

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Realistic Speech-Driven Facial Animation with GANs (2019, methodology): “CNN based GAN approaches have been used for speech to video approaches such as the one proposed in [44].”
The Creation and Detection of Deepfakes (2020, background): “e authors in [189] mitigate identity leakage by disentangling the speech and identity latent spaces using adversarial classiers.”
Deep Audio-visual Learning: A Survey (2020, methodology): “…Vougioukas et al. [98] Applied a temporal GAN for more temporal consistency Chen et al. [99] Applied optical ﬂow Generated lips only Zhou et al. [100] Disentangled information Lacked realism Zhu et al. [73] Asymmetric mutual information estimation to capture modality coherence Suffered from the…”
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis (2021, background): “Recently, many deep-learning-based techniques [35, 42, 10, 58, 7, 43, 48, 59, 21, 57] are proposed to learn the audio-to-face translation by generative adversarial networks (GANs).”
MEAD: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation (2020, methodology): “Zhou et al. [58] adopted a representation disentanglement framework to drive diﬀerent identities to utter the same speech contents.”
Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset (2021, background): “For more accurate lip-sync results, some works use the audio-mouth mutual information loss[51], audio-mouth correlation loss[5] and audio-visual disentangle learning[48].”
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild (2020, background): “Several works [5, 26, 28] train on datasets with a limited set of words such as GRID [10] (56 words), TIMIT [14] and LRW [8] (1000 words) which significantly hampers a model from learning the vast diversity of phoneme-viseme mappings in real videos [18].”
Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss (2019, methodology): “6) is conducted to investigate the visual qualities of our generated results compared with Chung et al. [3] and Zhou et al. [12].”
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (2021, methodology): “A number of previous studies [14, 72, 45, 44] leverage skip connections for better input identity preserving.”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, background): “On the other hand, subject-agnostic models [4,9,11,20,22,24,30,56,60,68,70,81,82,86,92,93,100] only need a single image of the target person, who is not seen during training, to synthesize arbitrary motions.”
What comprises a good talking-head video generation?: A Survey and Benchmark (2020, methodology): “Zhou et al. (2019) utilize the audio-visual retrieval protocol to evaluate the closeness between the audio and visual features.”
DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection (2020, unknown): “Another interesting approach was presented in [185].”
CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, background): “Conditional face video generation mainly including face reenactment [87,71,95,83] and talking face generation [98,10,102,99,38].”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, background): “Leveraging this link, cross-modal learning methods explore a range of interesting tasks: face reconstruction from audio [54], talking face generation [82], emotion recognition [5], speaker diarization [13], speech recognition [16], and speaker identification [50, 49, 39, 74, 17].”
Deep Self-Learning From Noisy Labels (2019, background): “Deep Neural Networks (DNNs) achieve impressive results on many computer vision tasks such as image recognition [13, 33, 34], semantic segmentation [22, 40, 24], object detection [5, 30, 27, 18] and cross modality tasks [20, 21, 41].”
FaceFormer: Speech-Driven 3D Facial Animation with Transformers (2021, unknown): “photorealistic talking-head animation [12, 15, 42, 52, 63, 67, 69].”
Discriminability Distillation in Group Representation Learning (2020, background): “Group representation learning which aims at formulating a unified representation has been proved efficient on various tasks [68,37,15,55,70].”
MakeItTalk: Speaker-Aware Talking Head Animation (2020, unknown): “Synthesizingplausibletalking-headshasnumerousapplications[Kim etal.2019;Zhouetal.2019].”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “[108] Temporal GAN Deep audio-video features ▪ LRW ▪MS-Celeb-1 M 256×256 ▪ lacks the ability to synthesize emotional facial expressions”