End-to-End Audiovisual Speech Recognition
By Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Feipeng Cai, Georgios Tzimiropoulos, Maja Pantic. IEEE International Conference on Acoustics, Speech, and Signal Processing, 2018
297 citations recorded by the source corpus.
Several end-to-end deep learning approaches have been recently presented which extract either audio or visual features from the input images or audio signals and perform speech recognition. However, research on end-to-end audiovisual models is very limited. In this work, we present an end-to-end audiovisual model based on residual networks and Bidirectional Gated Recurrent Units (BGRUs). To the best of our knowledge, this is the first audiovisual fusion model which simultaneously learns to extract features directly from the image pixels and audio waveforms and performs within-context word recognition on a large publicly available dataset (LRW). The model consists of two streams, one for each modality, which extract features directly from mouth regions and raw waveforms. The temporal dynamics in each stream/modality are modeled by a 2-layer BGRU and the fusion of multiple streams/modalities takes place via another 2-layer BGRU. A slight improvement in the classification rate over an end-to-end audio-only and MFCC-based model is reported in clean audio conditions and low levels of noise. In presence of high levels of noise, the end-to-end audiovisual model significantly outperforms both audio-only models.
Subfields: Cognitive Neuroscience, Signal Processing
View Paper: https://www.semanticscholar.org/paper/aadcf6136b9eaa0c96358e4351a070258a3bcc40
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Cognitive Neuroscience
- Jinyu Li
- Brais Martínez
- Israel Cohen
- Maja Pantic
- Andrew Zisserman
- Oriol Vinyals
- Triantafyllos Afouras
- Stavros Petridis
In Signal Processing
- Andrew Zisserman
- Joon Son Chung
- Jinyu Li
- Maja Pantic
- Triantafyllos Afouras
- Stavros Petridis
- Pingchuan Ma
- Dong Yu
Citations in the corpus
- Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “In contrast, it is very easy to build a multilingual E2E ASR system by simply taking a union of the token (e.g., character or subword) sets of all languages as the output token set and then training an E2E model on all the data [125–128].”
- End-To-End Audio-Visual Speech Recognition with Conformers (2021, methodology): “In particular, [24] was applied only to word classification while [17] was tested on a constrained environment.”
- An End-to-End Multimodal Voice Activity Detection Using WaveNet Encoder and Residual Networks (2019, background): “[41] proposed an end-to-end approach for audio-visual speech recognition.”
- Deep Audio-Visual Speech Recognition (2018, methodology): “The visual information can be used to improve the performance of ASR, particularly in environments with background noise [36], [38], [40].”
- Lipreading Using Temporal Convolutional Networks (2020, methodology): “The baseline corresponds to the model presented in [15].”
- The Conversation: Deep Audio-Visual Speech Enhancement (2018, methodology): “The enhancement method proposed here is complementary to lip reading [31, 32, 33], which has also been shown to improve ASR performance in noisy environments [34, 35].”