Audio-driven facial animation by joint end-to-end learning of pose and emotion

By Tero Karras, Timo Aila, Samuli Laine, Antti Herva, Jaakko Lehtinen. 2017

486 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Signal Processing

Citations in the corpus

VoxCeleb2: Deep Speaker Recognition (2018, background): “The dataset is audio-visual, so is also useful for a number of other applications, for example – visual speech synthesis [9, 10], speech separation [11, 12], † These authors contributed equally to this work. cross-modal transfer from face to voice or vice versa [13, 14] and training face…”
Executing your Commands via Motion Diffusion in Latent Space (2022, background): “Besides, MLD focuses on articulated human bodies, while there is also other work on faces [27, 6], hands [58, 37, 36] and even animal [60, 79] motion.”
Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss (2019, background): “However, recent researchers [3, 12, 17] tended to formulate video generation as a temporally independent image generation problem.”
State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications (2018, background): “To pro-c The approach of Karras et al. [KAL ∗ 17b] is also able to drive facial animations based on an audio stream.”
X2Face: A network for controlling face generation by using images, audio, and pose codes (2018, background): “Examples are controlling the mouth with speech [8, 38], controlling a head with audio and a known emotional state [16], and controlling body movement with music [36].”
Voxceleb: Large-scale speaker verification in the wild (2020, methodology): “The dataset is audio-visual, so is also useful for a number of other applications, for example, visual speech synthesis (Chung et al., 2017; Karras et al., 2017), speech separation (Afouras et al., 2018; Ephrat et al., 2018), cross-modal transfer from face to voice or vice versa (Nagrani et al.,…”
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis (2021, methodology): “Based on the developed 3D face reconstruction [19, 11, 50] and generative adversarial networks, more and more approaches are proposed by inter-mediately estimating 3D face shapes [22, 43, 55] or facial landmarks [56, 47].”
Realistic Speech-Driven Facial Animation with GANs (2019, methodology): “landmarks, visemes) and using computer graphics (CG) methods to generate realistic faces [19].”
MakeItTalk: Speaker-Aware Talking Head Animation (2020, methodology): “Many prior work resorted to subjective user studies [Cudeiro et al. 2019; Karras et al. 2017].”
MotionGPT: Human Motion as a Foreign Language (2023, unknown): “MotionGPT only utilizes motion on articulated human bodies, while many other works focus on faces [16, 4], hands [39, 22, 21] and even animal [40, 62] motion.”
What comprises a good talking-head video generation?: A Survey and Benchmark (2020, background): “[10] propose a network to synthesize 3D vertex by inferring the information from the audio signal and emotional state.”
Capture, Learning, and Synthesis of 3D Speaking Styles (2019, methodology): “Finally, while previous work has shown that models can be trained to create speaker-specific animations [14, 33], there are no generic methods that are speaker independent and that capture a variety of speaking styles.”
TEMOS: Generating diverse human motions from textual descriptions (2022, background): “While there is also work on facial motion generation [8, 11, 23, 43], here we focus on articulated human bodies.”
3D Morphable Face Models—Past, Present, and Future (2019, unknown): “Besides vision-based animation, there is extended work on audio-based control [Karras et al. 2017; Kshirsagar and Magnenat-Thalmann 2003; Taylor et al. 2017].”