Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis

By Prajwal K R, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, C. V. Jawahar. 2020

144 citations recorded by the source corpus.

Humans involuntarily tend to infer parts of the conversation from lip movements when the speech is absent or corrupted by external noise. In this work, we explore the task of lip to speech synthesis, i.e., learning to generate natural speech given only the lip movements of a speaker. Acknowledging the importance of contextual and speaker-specific cues for accurate lip-reading, we take a different path from existing works. We focus on learning accurate lip sequences to speech mappings for individual speakers in unconstrained, large vocabulary settings. To this end, we collect and release a large-scale benchmark dataset, the first of its kind, specifically to train and evaluate the single-speaker lip to speech task in natural settings. We propose a novel approach with key design choices to achieve accurate, natural lip to speech synthesis in such unconstrained scenarios for the first time. Extensive evaluation using quantitative, qualitative metrics and human evaluation shows that our method is four times more intelligible than previous works in this space.

Subfields: Artificial Intelligence, Pharmacy, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Pharmacy

Dong Yu

In Signal Processing

Citations in the corpus

An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, methodology): “[206] 2020 Raw pixels mel-scale CNN, LSTM 3 7 7 face spectrogram”
Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis (2022, methodology): “The results on Facestar and Lip2Wave are shown in Table 2.”