Capture, Learning, and Synthesis of 3D Speaking Styles

By Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, Michael J. Black. Computer Vision and Pattern Recognition, 2019

451 citations recorded by the source corpus.

Audio-driven 3D facial animation has been widely explored, but achieving realistic, human-like performance is still unsolved. This is due to the lack of available 3D datasets, models, and standard evaluation metrics. To address this, we introduce a unique 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio from 12 speakers. We then train a neural network on our dataset that factors identity from facial motion. The learned model, VOCA (Voice Operated Character Animation) takes any speech signal as input—even speech in languages other than English—and realistically animates a wide range of adult faces. Conditioning on subject labels during training allows the model to learn a variety of realistic speaking styles. VOCA also provides animator controls to alter speaking style, identity-dependent facial shape, and pose (i.e. head, jaw, and eyeball rotations) during animation. To our knowledge, VOCA is the only realistic 3D facial animation model that is readily applicable to unseen subjects without retargeting. This makes VOCA suitable for tasks like in-game video, virtual reality avatars, or any scenario in which the speaker, speech, or language is not known in advance. We make the dataset and model available for research purposes at http://voca.is.tue.mpg.de.

View Paper: https://www.semanticscholar.org/paper/91a1370d26ee903296bb4990a84c23f2fa8e8c83

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis (2021, background): “Moreover, existing methods suffer from several limitations, such as only rendering the mouth part [34, 36] or fixed by static head pose [28, 35, 10, 7], thus are not suitable for advanced talking head editing tasks, like pose-manipulation and background-replacement.”
FaceFormer: Speech-Driven 3D Facial Animation with Transformers (2021, methodology): “Alternatively, a wide variety of data-driven approaches [6, 17, 28, 31, 40, 48, 51, 53] has been proposed to produce 3D facial animation.”
I M Avatar: Implicit Morphable Head Avatars from Videos (2021, methodology): “We take FLAME expression parameters from the VOCA speech dataset [12] and head poses fitted from real videos.”
MakeItTalk: Speaker-Aware Talking Head Animation (2020, methodology): “Cudeiroetal.”
Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset (2021, methodology): “Several works use CNN-based[22, 37, 12] or LSTM-based[32, 30] framework to capture co-articulation effects.”
Learning an animatable detailed 3D face model from in-the-wild images (2020, background): “2019], Deng et al. [2019], FML [Tewari et al.”
3D Morphable Face Models—Past, Present, and Future (2019, unknown): “ help of 3DMMs. These include works that synthesize 4D videos from a static 3D mesh paired with semantic label information [Bolkart and Wuhrer 2015a], and from a static 3D mesh and audio information [Cudeiro et al. 2019]. 3.7 Publicly available models In Table 2 we list publicly available shape and/”
TEMOS: Generating diverse human motions from textual descriptions (2022, background): “While there is also work on facial motion generation [24,8,46,12], here we focus on articulated human bodies.”
GRAB: A Dataset of Whole-Body Human Grasping of Objects (2020, background): “Such whole-body grasping [28] has received much less attention [5,40] than single hand-object grasping [3,12,16,30,47].”