SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation

By Xiaodong Cun, Xuan Wang, Yong Zhang, Xiaodong Shen, Yu Guo, Ying Shan, Fei Wang. Computer Vision and Pattern Recognition, 2022

493 citations recorded by the source corpus.

Generating talking head videos through a face image and a piece of speech audio still contains many challenges. i.e., unnatural head movement, distorted expression, and identity modification. We argue that these issues are mainly caused by learning from the coupled 2D motion fields. On the other hand, explicitly using 3D information also suffers problems of stiff expression and incoherent video. We present SadTalker, which generates 3D motion coefficients (head pose, expression) of the 3DMM from audio and implicitly modulates a novel 3D-aware face render for talking head generation. To learn the realistic motion coefficients, we explicitly model the connections between audio and different types of motion coefficients individually. Precisely, we present ExpNet to learn the accurate facial expression from audio by distilling both coefficients and 3D-rendered faces. As for the head pose, we design PoseVAE via a conditional VAE to synthesize head motion in different styles. Finally, the generated 3D motion coefficients are mapped to the unsupervised 3D keypoints space of the proposed face render to synthesize the final video. We conducted extensive experiments to demonstrate the superiority of our method in terms of motion and video quality.11The code and demo videos are available at https://sadtalker.github.io.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/a7019f9a2ecb9983b66456ae32978e2574625980

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Ying Shan
2. Xintao Wang
3. Xiaodong Cun
4. Yong Zhang
5. Yaofang Liu
6. Haoxin Chen
7. Yang Liu
8. Raymond H. Chan

In Signal Processing

1. Ying Shan
2. Yong Zhang
3. Xiaodong Cun
4. Xintao Wang

Citations in the corpus

• PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding (2023, unknown): “Customized image generation related to humans [15, 30, 37, 56] has received considerable attention, giving rise to numerous applications, such as personalized portrait photos [41], image animation [77], and virtual try-on [67].”
• EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, background): “However, the quality of the generated videos is often low quality or can only work on a specific domain, e.g ., face [66] or landscape [50,63].”