A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild

By Prajwal K R, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, C. V. Jawahar. ACM Multimedia, 2020

1,210 citations recorded by the source corpus.

In this work, we investigate the problem of lip-syncing a talking face video of an arbitrary identity to match a target speech segment. Current works excel at producing accurate lip movements on a static image or videos of specific people seen during the training phase. However, they fail to accurately morph the lip movements of arbitrary identities in dynamic, unconstrained talking face videos, resulting in significant parts of the video being out-of-sync with the new audio. We identify key reasons pertaining to this and hence resolve them by learning from a powerful lip-sync discriminator. Next, we propose new, rigorous evaluation benchmarks and metrics to accurately measure lip synchronization in unconstrained videos. Extensive quantitative evaluations on our challenging benchmarks show that the lip-sync accuracy of the videos generated by our Wav2Lip model is almost as good as real synced videos. We provide a demo video clearly showing the substantial impact of our Wav2Lip model, and also publicly release the code, models, and evaluation benchmarks on our website.

Subfields: Computer Vision and Pattern Recognition, Pharmacy, Signal Processing

View Paper: https://www.semanticscholar.org/paper/9c160a71d3265eedaf7645c39be073c966f10433

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Pharmacy

In Signal Processing

Citations in the corpus

FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset (2021, methodology): “FSGAN or reenactmentmethod such asWav2Lip [29] to create deepfakes, as shown in Step 1–2.”
Deepfake Video Detection Using Convolutional Vision Transformer (2021, background): “Deepfake can also be created by using different techniques such as face swap [43], puppet-master [53], lip-sync [49, 47], facereenactment [14], synthetic image or video generation, and speech synthesis [48].”
Generative artificial intelligence: a systematic review and applications (2024, positive): “When observing the generated video frames produced by the StyleTalker in comparison to those generated by other audio-driven generation models [103, 138, 157], a notable distinction becomes evident.”
Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset (2021, methodology): “While many works[7, 6, 18, 13, 51, 5, 48, 29, 42, 49, 4] make great efforts to synthesize realistic-looking videos, the generation of high-resolution videos is still a challenge.”
FaceFormer: Speech-Driven 3D Facial Animation with Transformers (2021, methodology): “While aware of extensive 2D-based approaches [10, 11, 16, 18, 23, 29, 49, 59, 66, 70], we focus on animating a 3D model in this work.”
Multimodal Large Language Models: A Survey (2023, background): “Specifically, natural language generation technology impacts the quality of content in human-computer interactions, while computer vision technology affects the facial expressions and body movements of digital humans, such as lip synchronization [62].”
Data-Efficient Training Strategies for Neural TTS Systems (2020, methodology): “A demo video illustrating a couple of applications enabled by our TTS models such as translating lecture videos and public speeches [5, 16] is available here: Link1.”
HunyuanVideo: A Systematic Framework For Large Video Generative Models (2024, neutral): “Even earlier algorithms, like wav2lip [68] and DINet [97], concentrated on modifying the mouth region in the input video to achieve lip shape consistency with the audio.”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “The dataset is generated using real videos collected from YouTube and popular synthetic algorithms such as FSGAN [67], FaceSwap [66], Tacotron [233, 238], and Wave2Lip [111].”
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (2021, methodology): “A number of previous studies [14, 72, 45, 44] leverage skip connections for better input identity preserving.”