Neural Voice Puppetry: Audio-driven Facial Reenactment

By Justus Thies, Mohamed A. Elgharib, Ayush Kumar Tewari, Christian Theobalt, Matthias Nießner. European Conference on Computer Vision, 2019

453 citations recorded by the source corpus.

We present Neural Voice Puppetry, a novel approach for audio-driven facial video synthesis. Given an audio sequence of a source person or digital assistant, we generate a photo-realistic output video of a target person that is in sync with the audio of the source input. This audio-driven facial reenactment is driven by a deep neural network that employs a latent 3D face model space. Through the underlying 3D representation, the model inherently learns temporal stability while we leverage neural rendering to generate photo-realistic output frames. Our approach generalizes across different people, allowing us to synthesize videos of a target actor with the voice of any unknown source actor or even synthetic voices that can be generated utilizing standard text-to-speech approaches. Neural Voice Puppetry has a variety of use-cases, including audio-driven video avatars, video dubbing, and text-driven video synthesis of a talking head. We demonstrate the capabilities of our method in a series of audio- and text-based puppetry examples, including comparisons to state-of-the-art techniques and a user study.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/f5b0ece86e3b45f23a249e16691da6d71487cb37

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild (2020, background): “More recent works [13, 23] in this line directly generate images from speech representations and show exceptional generation quality for specific speakers which they have been trained upon.”
Regeneration Learning: A Learning Paradigm for Data Generation (2023, background): “…and melody in conditional melody generation (Ju et al. 2021; Wu et al. 2020; Dai et al. 2021; Zou et al. 2021; Di et al. 2021), speech and face images in talking-head video synthesis (Thies et al. 2020; Ji et al. 2021; Yi et al. 2020; Lahiri et al. 2021; Song et al. 2021; Ling et al. 2022).”
Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction (2020, methodology): “Our method is built upon a low-dimensional morphable model [7, 41] which is a building block of numerous facial reconstruction and animation approaches [14, 15, 46, 45, 40, 41, 5, 6, 21, 39, 38].”
Deep learning for deepfakes creation and detection: A survey (2019, unknown): “[54].”
Exploring Temporal Coherence for More General Video Face Forgery Detection (2021, background): “Currently, most face manipulation methods [31, 47, 48, 49, 53, 52] are specifically crafted at the image level.”
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (2021, background): “Thus the most plausible way is to leverage 3D models [33, 52, 8, 71] where the pose parameters and expression parameters are explicitly deﬁned [2].”
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis (2021, methodology): “In comparison, we extract the audio from the released demos of the two methods as the input of our framework (we assume the released demos as their best results since both of them did not provide pre-trained model), named as testset A (from Neural Voice Puppertry [43]) and test-set B (from Synthesiz”
MakeItTalk: Speaker-Aware Talking Head Animation (2020, background): “Our supplementary video also includes a comparison with the concurrent work by [Thies et al. 2020].”
Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection (2020, background): “Face manipulation methods in particular have raised considerable concerns due to their ability to alter a person’s identity [38, 30, 34], expression [52, 51, 44], or lip movements [50, 57, 56] to match the face in a given target video.”
Advances in Neural Rendering (2021, methodology): “But instead of using expression coefﬁcients, they map audio features extracted using Deep-Speech [HCC ∗ 14, TET ∗ 20] to a feature which serves as a conditioning to the MLP that represents the radiance ﬁeld.”
The Creation and Detection of Deepfakes (2020, methodology): “, [55, 154, 170]).”
FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset (2021, background): “Bush, which were generated as part of research [39].”
Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset (2021, background): “[39] leverage a pre-trained audio-to-mouth model to reduce the required footage to 2 ∼ 3 minutes.”