Large-Scale Visual Speech Recognition

By Brendan Shillingford, Yannis Assael, Matthew W. Hoffman, T. Paine, Cían Hughes, Utsav Prabhu, Hank Liao, Hasim Sak, Kanishka Rao, Lorrayne Bennett and 7 more. 2018

172 citations recorded by the source corpus.

This work presents a scalable solution to open-vocabulary visual speech recognition. To achieve this, we constructed the largest existing visual speech recognition dataset, consisting of pairs of text and video clips of faces speaking (3,886 hours of video). In tandem, we designed and trained an integrated lipreading system, consisting of a video processing pipeline that maps raw video to stable videos of lips and sequences of phonemes, a scalable deep neural network that maps the lip videos to sequences of phoneme distributions, and a production-level speech decoder that outputs sequences of words. The proposed system achieves a word error rate (WER) of 40.9% as measured on a held-out set. In comparison, professional lipreaders achieve either 86.4% or 92.9% WER on the same dataset when having access to additional types of contextual information. Our approach significantly improves on other lipreading approaches, including variants of LipNet and of Watch, Attend, and Spell (WAS), which are only capable of 89.8% and 76.8% WER respectively.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Natural Language Processing

In Signal Processing

Citations in the corpus

Group Normalization (2018, background): “GN also facilitates training joint speech and video networks (Shillingford et al. 2018) that are challenged by small batch sizes and variable lengths.”
End-To-End Audio-Visual Speech Recognition with Conformers (2021, methodology): “Shilling-ford et al. [27] proposed an improved version of the model called Vision to Phoneme (V2P) which predicts phoneme distributions, instead of characters, from video clips.”
Ava Active Speaker: An Audio-Visual Dataset for Active Speaker Detection (2019, methodology): “For applications that mine data corresponding to speaking faces (for tasks like synchronization [9], visual speech recognition [41], enhancement [1]), these models can be deployed without needing additional calibration and hand-tuning.”
Deep Audio-Visual Speech Recognition (2018, methodology): “[43], design a lip readingpipeline that uses a networkwhich outputs phoneme probabilities and is trained with CTC loss.”
Lipreading Using Temporal Convolutional Networks (2020, methodology): “Such works use either fully connected [7, 8, 9] or convolutional layers [10, 11, 12, 13] to extract features from the mouth region, and then feed them into a recurrent neural network or to attention [13, 14] / selfattention architectures [12].”
LRS3-TED: a large-scale dataset for visual speech recognition (2018, unknown): “In the recent years, there has been signiﬁcant progress [1, 2, 3, 4] in the performance of automated lip reading due to the application of deep neural network models and the availability of large scale datasets.”
Artificial intelligence in the creative industries: a review (2020, background): “Google proposed a full visual-speech recognition system that maps videos of lips to sequences of words using spatiotemporal CNNs and LSTMs [174].”