Combining Residual Networks with LSTMs for Lipreading

By Themos Stafylakis, Georgios Tzimiropoulos. Interspeech, 2017

353 citations recorded by the source corpus.

We propose an end-to-end deep learning architecture for word-level visual speech recognition. The system is a combination of spatiotemporal convolutional, residual and bidirectional Long Short-Term Memory networks. We train and evaluate it on the Lipreading In-The-Wild benchmark, a challenging database of 500-size target-words consisting of 1.28sec video excerpts from BBC TV broadcasts. The proposed network attains word accuracy equal to 83.0, yielding 6.8 absolute improvement over the current state-of-the-art, without using information about word boundaries during training or testing.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/4afdb836301c0233bb8cf0d8a33212ac0c113381

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, background): “performing reasonably well in both cases exist [18], [41], [205], [227], [233], [234].”
The Conversation: Deep Audio-Visual Speech Enhancement (2018, methodology): “We pre-train the spatio-temporal visual front-end on a wordlevel lip reading task, following [33].”
Large-Scale Visual Speech Recognition (2018, methodology): “Often the approaches focus on single-word classiﬁcation (Hinton et al., 2012; Chung & Zisserman, 2016a; Wand et al., 2016; Stafylakis & Tzimiropoulos, 2017; Ngiam et al., 2011; Sui et al., 2015; Ninomiya et al., 2015; Petridis & Pantic, 2016; Petridis et al., 2017; Noda et al., 2014; Koller et al.,…”
End-to-End Audiovisual Speech Recognition (2018, methodology): “The output of the ResNet-18 is fed to a 2-layer BGRU which consists of 1024 cells in each layer (using the same architecture as in [13]).”
Deep Audio-Visual Speech Recognition (2018, methodology): “architecture of [45] to learn representations from raw”
Deep Audio-visual Learning: A Survey (2020, methodology): “They either leverage the fully connected layers and LSTM to extract features and model the temporal information [56, 57] or use a 3D convolutional layer followed by a combination of CNNs and LSTMs [58, 68] .”
End-To-End Audio-Visual Speech Recognition with Conformers (2021, methodology): “For the visual stream, we use a modiﬁed ResNet-18 [11, 28] in which the ﬁrst convolutional layer is replaced by a 3D convolutional layer with a kernel size 5 × 7 × 7 .”
MMTM: Multimodal Transfer Module for CNN Fusion (2019, methodology): “Following [50], we discard redundant visual information by cropping the mouth region via facial landmarks obtained from Facial Alignment Network [71].”
Talking Face Generation by Adversarially Disentangled Audio-Visual Representation (2018, background): “In recent years, this field develop quickly with the usage of convolutional neural networks (CNNs) and recurrent neural networks (RNNs) for end-to-end word-level (Chung and Zisserman 2016a; Stafylakis and Tzimiropoulos 2017), sentencelevel (Assael et al.”
Lipreading Using Temporal Convolutional Networks (2020, methodology): “Such works use either fully connected [7, 8, 9] or convolutional layers [10, 11, 12, 13] to extract features from the mouth region, and then feed them into a recurrent neural network or to attention [13, 14] / selfattention architectures [12].”
LRS3-TED: a large-scale dataset for visual speech recognition (2018, neutral): “In the recent years, there has been signiﬁcant progress [1, 2, 3, 4] in the performance of automated lip reading due to the application of deep neural network models and the availability of large scale datasets.”
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction (2022, neutral): “The image encoder is a modified ResNet-18, which has been used in prior work (Ma et al., 2021b; Martinez et al., 2020; Stafylakis & Tzimiropoulos, 2017). in Makino et al. (2019).”
Learning to lip read words by watching videos (2018, background): “Extensions could include lip reading of profile views, and varying the architecture (in terms of depth, 3D CNNs etc) to improve performance – there is already evidence that there are benefits of using deeper architectures [48] on our released dataset.”