FaceFormer: Speech-Driven 3D Facial Animation with Transformers
By Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, Taku Komura. Computer Vision and Pattern Recognition, 2021
326 citations recorded by the source corpus.
Speech-driven 3D facial animation is challenging due to the complex geometry of human faces and the limited availability of 3D audio-visual data. Prior works typically focus on learning phoneme-level features of short audio windows with limited context, occasionally resulting in inaccurate lip movements. To tackle this limitation, we propose a Transformer-based autoregressive model, Face-Former, which encodes the long-term audio context and autoregressively predicts a sequence of animated 3D face meshes. To cope with the data scarcity issue, we integrate the self-supervised pre-trained speech representations. Also, we devise two biased attention mechanisms well suited to this specific task, including the biased cross-modal multi-head (MH) attention and the biased causal MH self-attention with a periodic positional encoding strategy. The former effectively aligns the audio-motion modalities, whereas the latter offers abilities to generalize to longer audio sequences. Extensive experiments and a perceptual user study show that our approach outperforms the existing state-of-the-arts. The code and the video are available at: https://evelynfan.github.io/audio2face/
Subfields: Computer Vision and Pattern Recognition, Signal Processing
View Paper: https://www.semanticscholar.org/paper/dd23991294bf53c6301ab79fa49752e6458d2eab
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Gul Varol
- Michael J. Black
- Mathis Petrovich
- Luc Van Gool
- Jitendra Malik
- Cordelia Schmid
- Andrew Zisserman
- Ming-Hsuan Yang
In Signal Processing
Citations in the corpus
- TEMOS: Generating diverse human motions from textual descriptions (2022, background): “While there is also work on facial motion generation [24,8,46,12], here we focus on articulated human bodies.”
- Multimodal Learning With Transformers: A Survey (2022, methodology): “[39] Y. Fan, Z. Lin, J. Saito, W. Wang, and T. Komura, “FaceFormer: Speechdriven 3D facial animation with transformers,” 2021, arXiv:2112.05329.”
- SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation (2022, unknown): “Zoomed Regions”