MakeItTalk: Speaker-Aware Talking Head Animation

By Yang Zhou, Dingzeyu Li, Xintong Han, Evangelos Kalogerakis, Eli Shechtman, Jose I. Echevarria. arXiv.org, 2020

551 citations recorded by the source corpus.

We present a method that generates expressive talking heads from a single facial image with audio as the only input. In contrast to previous approaches that attempt to learn direct mappings from audio to raw pixels or points for creating talking faces, our method first disentangles the content and speaker information in the input audio signal. The audio content robustly controls the motion of lips and nearby facial regions, while the speaker information determines the specifics of facial expressions and the rest of the talking head dynamics. Another key component of our method is the prediction of facial landmarks reflecting speaker-aware dynamics. Based on this intermediate representation, our method is able to synthesize photorealistic videos of entire talking heads with full range of motion and also animate artistic paintings, sketches, 2D cartoon characters, Japanese mangas, stylized caricatures in a single unified framework. We present extensive quantitative and qualitative evaluation of our method, in addition to user studies, demonstrating generated talking heads of significantly higher quality compared to prior state-of-the-art.

View Paper: https://www.semanticscholar.org/paper/7a7c4e666e5dd8a489770643b72fd99cf708f7c1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Chen Change Loy
2. Xiaogang Wang
3. Ziwei Liu
4. Xiaodong Cun
5. Ping Luo
6. Fei Wang
7. Wayne Wu
8. Yi Yang

Citations in the corpus

• Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (2021, background): “However, the lip sync score of Wav2Lip [44] is the same as ours, outperforming MakeitTalk [75].”
• SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation (2022, methodology): “Their methods mainly introduce 2D motion fields by landmarks [50] and latent warping [37,38].”
• Generative artificial intelligence: a systematic review and applications (2024, unknown): “When observing the generated video frames produced by the StyleTalker in comparison to those generated by other audio-driven generation models [103, 138, 157], a notable distinction becomes evident.”
• Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset (2021, methodology): “Current best work[49] just generate videos with 256 × 256 resolution(see Figure10(e) for example), however, directly employing their model on 512 × 512 image will get blurry results (see Figure10(f) for example).”