LipNet: End-to-End Sentence-level Lipreading

By Yannis Assael, Brendan Shillingford, Shimon Whiteson, Nando de Freitas. 2016

482 citations recorded by the source corpus.

Lipreading is the task of decoding text from the movement of a speaker's mouth. Traditional approaches separated the problem into two stages: designing or learning visual features, and prediction. More recent deep lipreading approaches are end-to-end trainable (Wand et al., 2016; Chung & Zisserman, 2016a). However, existing work on models trained end-to-end perform only word classification, rather than sentence-level sequence prediction. Studies have shown that human lipreading performance increases for longer words (Easton & Basala, 1982), indicating the importance of features capturing temporal context in an ambiguous communication channel. Motivated by this observation, we present LipNet, a model that maps a variable-length sequence of video frames to text, making use of spatiotemporal convolutions, a recurrent network, and the connectionist temporal classification loss, trained entirely end-to-end. To the best of our knowledge, LipNet is the first end-to-end sentence-level lipreading model that simultaneously learns spatiotemporal visual features and a sequence model. On the GRID corpus, LipNet achieves 95.2% accuracy in sentence-level, overlapped speaker split task, outperforming experienced human lipreaders and the previous 86.4% word-level state-of-the-art accuracy (Gergen et al., 2016).

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/291c0e453503a704c0fd932a067ca054cc7edad6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Dong Yu
3. Tara N. Sainath
4. Yoshua Bengio
5. Hasim Sak
6. Quoc V. Le
7. Andrew W. Senior
8. Kanishka Rao

In Signal Processing

1. Jinyu Li
2. Dong Yu
3. Tara N. Sainath
4. Hasim Sak
5. Andrew W. Senior
6. Kanishka Rao
7. Shinji Watanabe
8. Hank Liao

Citations in the corpus

• What comprises a good talking-head video generation?: A Survey and Benchmark (2020, background): “Although there are many lipreading networks proposed in recent years, they are either limited to frontal face videos (Assael et al. 2016), or publicly unavailable (Zhang et al.”
• LSTM-Based EEG Classification in Motor Imagery Tasks (2018, unknown): “[14] can achieve an accuracy of 95.”
• End-To-End Audio-Visual Speech Recognition with Conformers (2021, methodology): “In VSR, Assael et al. [4] developed the first end-to-end network based on 3D convolution with Gated Recurrent Units (GRUs) for recognising visual speech on GRID [6].”
• Large-Scale Visual Speech Recognition (2018, methodology): “However, these approaches have often been limited to narrow vocabularies, and relatively small datasets [1, 4, 6].”
• Lip Movements Generation at a Glance (2018, background): “Observing that speech is highly correlated with lip movements even across identities, a concept grounds lip reading [1,7], the core of our paper is to explore the best modeling of such correlations in building and training a lip movement generator network.”
• An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, background): “performing reasonably well in both cases exist [18], [41], [205], [227], [233], [234].”
• Realistic Speech-Driven Facial Animation with GANs (2019, methodology): “We use the LipNet model [3], which surpasses the performance of human lip-readers on the GRID dataset.”