LRS3-TED: a large-scale dataset for visual speech recognition

By Triantafyllos Afouras, Joon Son Chung, Andrew Zisserman. arXiv.org, 2018

614 citations recorded by the source corpus.

This paper introduces a new multi-modal dataset for visual and audio-visual speech recognition. It includes face tracks from over 400 hours of TED and TEDx videos, along with the corresponding subtitles and word alignment boundaries. The new dataset is substantially larger in scale compared to other public datasets that are available for general research.

Subfields: Computer Vision and Pattern Recognition, Language and Linguistics, Signal Processing

View Paper: https://www.semanticscholar.org/paper/a46174aa635759070984ed7062c9402695bce830

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Language and Linguistics

In Signal Processing

Citations in the corpus

Deep Audio-Visual Speech Recognition (2018, positive): “This model also sets a baseline for LRS3-TED at 58.9 percent.”
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, background): “The trend of collecting larger datasets has continued in subsequent collections which consist of materials from British television programs [8], [39], [41], generic YouTube videos [38], [185], TED talks [9], [55], movies [217] or lectures [55], [244].”
Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis (2020, background): “The current datasets for lip to speech (or) text are at the two opposite ends of the spectrum: (i) small, constrained narrow vocabulary like GRID [7], TCD-TIMIT [13] or (ii) unconstrained, open vocabulary multi-speaker like LRS2 [2], LRW [6] and LRS3 [3].”
Is Someone Speaking?: Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection (2021, positive): “We leverage two large-scale audio-visual datasets in the wild, LRS3 [3] and VoxCeleb2 [16], to form a new ASD dataset, named “TalkSet”, that covers all valid ASD conditions.”
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild (2020, methodology): “p region, we found that lip landmarks can be quite inaccurate on generated faces. Thus, there is a need for a metric that is designed specifically for measuring lip-sync errors. LRW [8] LRS2 [1] LRS3 [3] Method LSE-D ↓ LSE-C ↑ FID ↓ LSE-D ↓ LSE-C ↑ FID ↓ LSE-D ↓ LSE-C ↑ FID ↓ Speech2Vid [17] 13.14 1”
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction (2022, positive): “(Afouras et al., 2018b) is the largest publicly available sentence-level lip reading dataset to date.”
Large-Scale Visual Speech Recognition (2018, methodology): “First, we evaluated performance on a subset of the LRS3-TED test set ﬁltered according to the same protocol used to construct LSVSR, by removing instances with larger face angles and shorter clips ( Filtered Test ).”
Survey on Deep Neural Networks in Speech and Vision Systems (2019, methodology): “A dataset called LRS3-TED has been compiled for visual speech recognition [151].”
End-To-End Audio-Visual Speech Recognition with Conformers (2021, methodology): “The language model is trained by combining the training transcriptions of LibriSpeech (960h) [21], pre-training and training sets of LRS2 [5] and LRS3 [3], with a total of 16.2 million words.”
Self-Supervised Learning of Audio-Visual Objects from Video (2020, methodology): “To assess the robustness of our method to large camera and speaker motions, we used optical ﬂow to rank videos by the amount of motion and obtained high motion subsets for the validation set of LRS2 and test set of LRS3.”
Multimodal SpeakerBeam: Single Channel Target Speech Extraction with Audio-Visual Speaker Clues (2019, methodology): “To evaluate the effectiveness of our proposed method, we created a simulation dataset of speech mixtures based on the Lip Reading Sentences 3 (LRS3-TED) audio-visual corpus [16].”
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement (2022, methodology): “The ﬁrst one is Lip Reading Sentences 3 (LRS3) [2], which is a clean dataset based on TED talk videos.”