What comprises a good talking-head video generation?: A Survey and Benchmark

By Lele Chen, Guofeng Cui, Ziyi Kou, Haitian Zheng, Chenliang Xu. 2020

74 citations recorded by the source corpus.

Over the years, performance evaluation has become essential in computer vision, enabling tangible progress in many sub-fields. While talking-head video generation has become an emerging research topic, existing evaluations on this topic present many limitations. For example, most approaches use human subjects (e.g., via Amazon MTurk) to evaluate their research claims directly. This subjective evaluation is cumbersome, unreproducible, and may impend the evolution of new research. In this work, we present a carefully-designed benchmark for evaluating talking-head video generation with standardized dataset pre-processing strategies. As for evaluation, we either propose new metrics or select the most appropriate ones to evaluate results in what we consider as desired properties for a good talking-head video, namely, identity preserving, lip synchronization, high video quality, and natural-spontaneous motion. By conducting a thoughtful analysis across several state-of-the-art talking-head generation approaches, we aim to uncover the merits and drawbacks of current methods and point out promising directions for future work. All the evaluation code is available at: this https URL.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey on Neural Speech Synthesis (2021, methodology): “Due to page limitations, we only reviewed core algorithms of TTS; readers can refer to other papers for TTS related problems and applications, such as voice conversion [302], singing voice synthesis [112, 211, 35], talking face synthesis [36], etc.”
Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset (2021, background): “Table 1 illustrates some common audio-visual datasets (all available datasets are listed in [3]).”
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (2021, background): “Driving talking faces with audio input [7, 78] has long been important research interest in both computer vision and graphics, where structural information and stitching techniques play a crucial role [4, 3, 57, 56, 76].”