Lipreading with long short-term memory

By Michael Wand, Jan Koutník, J. Urgen Schmidhuber. 2016

234 citations recorded by the source corpus.

Lipreading, i.e. speech recognition from visual-only recordings of a speaker's face, can be achieved with a processing pipeline based solely on neural networks, yielding significantly better accuracy than conventional methods. Feedforward and recurrent neural network layers (namely Long Short-Term Memory; LSTM) are stacked to form a single structure which is trained by back-propagating error gradients through all the layers. The performance of such a stacked network was experimentally evaluated and compared to a standard Support Vector Machine classifier using conventional computer vision features (Eigenlips and Histograms of Oriented Gradients). The evaluation was performed on data from 19 speakers of the publicly available GRID corpus. With 51 different words to classify, we report a best word accuracy on held-out evaluation speakers of 79.6% using the end-to-end neural network-based solution (11.6% improvement over the best feature-based solution evaluated).

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Signal Processing

Citations in the corpus

Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences (2016, background): “15% compares favorably against lipreading-focused state-of-the-art approaches [28] while avoiding manually-crafted features.”
Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis (2020, background): “In this space as well, several works [6, 28, 37, 38] are limited to narrow vocabularies and small datasets, however, unlike lip to speech, there have been multiple works [2, 5]”
LipNet: End-to-End Sentence-level Lipreading (2016, methodology): “To evaluate LipNet, we compare its performance to that of three hearing-impaired people who can lipread, as well as three ablation models inspired by recent state-of-the-art work (Chung & Zisserman, 2016a; Wand et al., 2016).”
End-to-End Audiovisual Speech Recognition (2018, methodology): “In the first one, fully connected layers are used to extract features and LSTM layers model the temporal dynamics of the sequence [10, 11].”
Large-Scale Visual Speech Recognition (2018, methodology): “…most works still focus on single-word classiﬁcation, either by learning visual-only representations (Hinton et al., 2012; Chung & Zisserman, 2016a; Wand et al., 2016; Stafylakis & Tzimiropoulos, 2017; Wand & Schmidhuber, 2017), multimodal audio-visual representations (Ngiam et al., 2011; Sui et…”
Lipreading Using Temporal Convolutional Networks (2020, methodology): “Such works use either fully connected [7, 8, 9] or convolutional layers [10, 11, 12, 13] to extract features from the mouth region, and then feed them into a recurrent neural network or to attention [13, 14] / selfattention architectures [12].”
Lip Reading Sentences in the Wild (2016, methodology): “The evaluation follows the standard protocol of [38] and [2] – the data is randomly divided into train, validation and test sets, where the latter contains 255 utterances for each speaker.”
Deep Audio-Visual Speech Recognition (2018, methodology): “[50] use an LSTM with HOG input features to recognise short phrases.”
Deep Audio-visual Learning: A Survey (2020, methodology): “[57] presented a word-level lip-reading system using LSTM.”