End-To-End Audio-Visual Speech Recognition with Conformers

By Pingchuan Ma, Stavros Petridis, Maja Pantic. IEEE International Conference on Acoustics, Speech, and Signal Processing, 2021

311 citations recorded by the source corpus.

In this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner. In particular, the audio and visual encoders learn to extract features directly from raw pixels and audio waveforms, respectively, which are then fed to conformers and then fusion takes place via a Multi-Layer Perceptron (MLP). The model learns to recognise characters using a combination of CTC and an attention mechanism. We show that end-to-end training, instead of using pre-computed visual features which is common in the literature, the use of a conformer, instead of a recurrent network, and the use of a transformer-based language model, significantly improve the performance of our model. We present results on the largest publicly available datasets for sentence-level speech recognition, Lip Reading Sentences 2 (LRS2) and Lip Reading Sentences 3 (LRS3), respectively. The results show that our proposed models raise the state-of-the-art performance by a large margin in audio-only, visual-only, and audio-visual experiments.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/fed1159caec29e0d1996b51984cff81d98185d2f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yossi Adi
2. Bowen Shi
3. Tal Remez
4. Wei-Ning Hsu
5. Jacob Donley
6. James R. Glass
7. Gabriel Synnaeve
8. Jade Copet

In Signal Processing

1. Bowen Shi
2. Wei-Ning Hsu
3. Yossi Adi
4. Tal Remez
5. Jacob Donley
6. James R. Glass
7. David F. Harwath
8. Gabriel Synnaeve

Citations in the corpus

• Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction (2022, unknown): “Unlike Ma et al. (2021a) and Afouras et al. (2020), our model is trained from scratch and encouraged to learn contextualized representations with a masked prediction task.”
• ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement (2022, methodology): “The video encoder is a ResNet-18 [35] model, while the audio-encoder is simply a linear projection.”