Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset

By Zhimeng Zhang, Lincheng Li, Yu Ding. Computer Vision and Pattern Recognition, 2021

590 citations recorded by the source corpus.

One-shot talking face generation should synthesize high visual quality facial videos with reasonable animations of expression and head pose, and just utilize arbitrary driving audio and arbitrary single face image as the source. Current works fail to generate over 256×256 resolution realistic-looking videos due to the lack of an appropriate high-resolution audio-visual dataset, and the limitation of the sparse facial landmarks in providing poor expression details. To synthesize high-definition videos, we build a large in-the-wild high-resolution audio-visual dataset and propose a novel flow-guided talking face generation framework. The new dataset is collected from youtube and consists of about 16 hours 720P or 1080P videos. We leverage the facial 3D morphable model (3DMM) to split the framework into two cascaded modules instead of learning a direct mapping from audio to video. In the first module, we propose a novel animation generator to produce the movements of mouth, eyebrow and head pose simultaneously. In the second module, we transform animation into dense flow to provide more expression details and carefully design a novel flow-guided video generator to synthesize videos. Our method is able to produce high-definition videos and outperforms state-of-the-art works in objective and subjective comparisons*.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c9e8cb4bb2c8e5d6dd4e631fc347f7f1415f2150

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xiaodong Cun
2. Yong Zhang
3. Kaitao Song
4. Ying Shan
5. Yichong Leng
6. Jiang Bian
7. Zeqian Ju
8. Hujun Bao

Citations in the corpus

• PromptTTS 2: Describing and Generating Voices with Text Prompt (2023, methodology): “For this preliminary experiment, we utilize the HDTF dataset (Zhang et al., 2021), a high-resolution dataset designed for talking face generation.”
• AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis (2021, background): “Recently, many deep-learning-based techniques [35, 42, 10, 58, 7, 43, 48, 59, 21, 57] are proposed to learn the audio-to-face translation by generative adversarial networks (GANs).”
• SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation (2022, methodology): “Comparison with the state-of-the-art method on HDTF dataset [13] with cross-identity setting.”
• Generative artificial intelligence: a systematic review and applications (2024, unknown): “6 DaGAN++ vs other state of art model on HDTF dataset [154], Source: [48] talking head video that align faithfully with the given audio input.”