Deep Audio-Visual Speech Recognition

By Triantafyllos Afouras, Joon Son Chung, Andrew W. Senior, Oriol Vinyals, Andrew Zisserman. 2018

889 citations recorded by the source corpus.

The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem – unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Meanwhile, visual information is increasingly used in historically audio-only tasks like speech transcription, voice recognition, audio spatialization [5,80,104,161], speaker diarization [10,83], and source separation [57,78,82].”
Survey on Deep Neural Networks in Speech and Vision Systems (2019, background): “Afouras et al. [141] use 3D CNN to capture spatio-temporal information of the face, and a transformer self-attention module guides the network for speech extraction from the extracted convolutional features.”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, methodology): “Mandarin [35], TCD-TIMIT [31], CUAVE [59], LRS23 [1]: We evaluate on these four standard benchmark datasets to compare our model with a series of state-of-theart audio-visual speech separation and enhancement methods in Sec.”
Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “In contrast, it is very easy to build a multilingual E2E ASR system by simply taking a union of the token (e.g., character or subword) sets of all languages as the output token set and then training an E2E model on all the data [125–128].”
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild (2020, methodology): “ideos. Also, it should not be fine-tuned further on the generated frames like it is done in LipGAN. One such network that has been used to correct lip-sync errors for creating large lip-sync datasets [1, 3] is the SyncNet [9] model. We propose to adapt and train a modified version of SyncNet [9] for”
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction (2022, neutral): “We conduct experiments on two datasets: LRS3 (Afouras et al., 2018b) with 433 hours of transcribed English videos and VoxCeleb2 (Chung et al., 2018) with 2442 hours of unlabeled multilingual videos.”
Lipreading Using Temporal Convolutional Networks (2020, methodology): “Such works use either fully connected [7, 8, 9] or convolutional layers [10, 11, 12, 13] to extract features from the mouth region, and then feed them into a recurrent neural network or to attention [13, 14] / selfattention architectures [12].”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “Futhermore, multi-modal learning has been used for several applications such as lip reading [4, 20, 21], audio-visual source separation and localization [3, 7, 28, 29, 39, 77, 87, 91, 113, 114], speech recognition [2, 73, 84], efficient inference [30, 56], egocentric action recognition [51] and navi”
Is Someone Speaking?: Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection (2021, neutral): “V-TCN Conv1D denotes point-wise addition.”
Self-Supervised Learning of Audio-Visual Objects from Video (2020, background): “LRS2 [1] and LRS3 [3] are audio-visual speech datasets containing 224 and 475 hours of videos respectively, along with ground truth face tracks of the speakers.”
Deep Audio-visual Learning: A Survey (2020, methodology): “To solve this problem, Afouras et al. [61] proposed a model for performing speech recognition tasks.”
End-To-End Audio-Visual Speech Recognition with Conformers (2021, methodology): “Several audio-visual approaches have been recently presented where pre-computed visual or audio features are used [1, 19, 25, 29, 34].”
Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation (2020, methodology): “To help our translation networks with sign language understanding and to achieve CSLR, we introduce a Sign Language Recognition Transformer (SLRT), an encoder transformer model trained using a CTC loss [2], to predict sign gloss sequences.”
Multimodal Learning With Transformers: A Survey (2022, background): “recognition [134], video & text for Referring Video Object Segmentation (RVOS) [221], source code & comment & data flow [44], image & text for retrieval [222].”
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, background): “The trend of collecting larger datasets has continued in subsequent collections which consist of materials from British television programs [8], [39], [41], generic YouTube videos [38], [185], TED talks [9], [55], movies [217] or lectures [55], [244].”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “vision can also be combined with speech to perform audio-visual speech recognition [53]–[55], speaker recognition [56]–[58],”
A General Survey on Attention Mechanisms in Deep Learning (2022, methodology): “For example, in [79], the self-attention Transformer architecture is used to create a model that can recognize phrases from audio and by lip-reading from a video.”