A deep learning approach for generalized speech animation

By Sarah L. Taylor, Taehwan Kim, Yisong Yue, Moshe Mahler, James Krahe, Anastasio Garcia Rodriguez, Jessica K. Hodgins, Iain Matthews. ACM Transactions on Graphics, 2017

320 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Signal Processing

View Paper: https://www.semanticscholar.org/paper/cc63b9cf84b1fb0b3eca84372919f74a40b7c132

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Dacheng Tao
4. Jitendra Malik
5. Michael J. Black
6. Xiaogang Wang
7. Chen Change Loy
8. Andrew Zisserman

In Control and Systems Engineering

1. Michael J. Black
2. Christian Theobalt
3. Timo Bolkart
4. Chaitanya Ahuja
5. Maja Pantic
6. Hujun Bao
7. Michael Zollhöfer
8. Yisong Yue

In Signal Processing

1. Louis-philippe Morency
2. Yu Ding
3. Lincheng Li
4. Zhimeng Zhang
5. Stavros Petridis
6. Maja Pantic
7. Chaitanya Ahuja
8. Hujun Bao

Citations in the corpus

• Realistic Speech-Driven Facial Animation with GANs (2019, methodology): “A typical example of this, proposed in [35], uses a deep neural network (DNN) to transform a phoneme sequence into a sequence of shapes for the lower half of the face.”
• Generative Multi-Agent Behavioral Cloning (2018, background): “, 2011), video & motion capture (Suwajanakorn et al., 2017; Taylor et al., 2017; Xue et al., 2016), navigation & driving (Ziebart et al.”
• Text-based editing of talking-head video (2019, methodology): “Several related methods produce animation curves for speech [Edwards et al. 2016; Taylor et al. 2017; Zhou et al. 2018].”
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision (2018, background): “…2014; Yue et al., 2014; Zheng et al., 2016; Le et al., 2017), video games (Ross et al., 2011), video & motion capture (Suwajanakorn et al., 2017; Taylor et al., 2017; Xue et al., 2016), navigation & driving (Ziebart et al., 2009; Zhang & Cho, 2017; Li et al., 2017), laboratory animal behaviors…”
• Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset (2021, methodology): “Several works use CNN-based[22, 37, 12] or LSTM-based[32, 30] framework to capture co-articulation effects.”
• Language2Pose: Natural Language Grounded Pose Forecasting (2019, methodology): “[32] use a data driven approach to model facial animation, while upper body pose forecasting conditioned on speech inputs has been tackled by Takeuchi et. al.”
• MakeItTalk: Speaker-Aware Talking Head Animation (2020, background): “With the increasing power of GPUs, we have seen prolific progress on end-to-end learning from audio to video frames, such as predicting lip movement [Chen et al. 2018; Taylor et al. 2017], generating full faces with GANs [Song et al.”
• AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis (2021, background): “Moreover, existing methods suffer from several limitations, such as only rendering the mouth part [34, 36] or fixed by static head pose [28, 35, 10, 7], thus are not suitable for advanced talking head editing tasks, like pose-manipulation and background-replacement.”
• Capture, Learning, and Synthesis of 3D Speaking Styles (2019, methodology): “Several works also aim at animating artist designed character rigs [20, 21, 31, 32, 46, 52, 53, 54, 70].”