Lip Reading Sentences in the Wild

By Joon Son Chung, Andrew W. Senior, Oriol Vinyals, Andrew Zisserman. Computer Vision and Pattern Recognition, 2016

931 citations recorded by the source corpus.

The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem – unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) a Watch, Listen, Attend and Spell (WLAS) network that learns to transcribe videos of mouth motion to characters, (2) a curriculum learning strategy to accelerate training and to reduce overfitting, (3) a Lip Reading Sentences (LRS) dataset for visual speech recognition, consisting of over 100,000 natural sentences from British television. The WLAS model trained on the LRS dataset surpasses the performance of all previous work on standard lip reading benchmark datasets, often by a significant margin. This lip reading performance beats a professional lip reader on videos from BBC television, and we also demonstrate that if audio is available, then visual information helps to improve speech recognition performance.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/bed6d0097df1e9ac82f789f6da268cdb3dd65bc3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, methodology): “While we study alignment for self-supervised learning, it has also been studied as an end in itself [20,21,22] e.g. in lip-reading applications [23].”
Perfect Match: Improved Cross-modal Embeddings for Audio-visual Synchronisation (2018, methodology): “The LRS2 dataset contains 96,318 clips for training, and 1,243 for test.”
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, background): “The trend of collecting larger datasets has continued in subsequent collections which consist of materials from British television programs [8], [39], [41], generic YouTube videos [38], [185], TED talks [9], [55], movies [217] or lectures [55], [244].”
LRS3-TED: a large-scale dataset for visual speech recognition (2018, positive): “The pipeline is based on the methods described in [1, 6], but we give a brief sketch of the method here.”
Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis (2020, background): “In this space as well, several works [6, 28, 37, 38] are limited to narrow vocabularies and small datasets, however, unlike lip to speech, there have been multiple works [2, 5]”
Neural Sign Language Translation (2018, methodology): “Similar attention based approaches have been applied to various Computer Vision tasks, such as image captioning [61], lip reading [13] and action recognition [19].”
Learning Weakly Supervised Multimodal Phoneme Embeddings (2017, background): “Other studies tackle the automatic lipreading task such as WLAS a r X i v : 1704 .”
Lip Movements Generation at a Glance (2018, methodology): “We use Dlib [18], a HOG-based facial landmarks detector, which is also widely used in lip-movement generation task and other related works[29,28], to detect lip landmarks on v̂ and v, and mark them as LF and LR, respectively.”
Deep Audio-Visual Speech Recognition (2018, methodology): “LRS [12] y BBC Train-val 01/2010 - 02/2016 - 106k 705k 17k 68 Test 03/2016 - 09/2016 - 12k 77k 6,882 7.”
Talking Face Generation by Adversarially Disentangled Audio-Visual Representation (2018, background): “In recent years, this ﬁeld develop quickly with the usage of convolutional neural networks (CNNs) and recurrent neural networks (RNNs) for end-to-end word-level [9, 25], sentence-level [4, 12], and multi-view [8] lip reading.”
X2Face: A network for controlling face generation by using images, audio, and pose codes (2018, background): “Examples are controlling the mouth with speech [8, 38], controlling a head with audio and a known emotional state [16], and controlling body movement with music [36].”
Squeeze-and-Excitation Networks (2017, background): “understanding in images [9], [54], image captioning [55], [56] and lip reading [57].”
Towards Better Decoding and Language Model Integration in Sequence to Sequence Models (2016, background): “Especially, Sequence-to-Sequence (seq2seq) neural networks have proved to be very successful on complex transduction tasks, such as machine translation [8, 9, 10], speech recognition [11, 12, 13], or lip-reading [14].”
You said that? (2017, methodology): “[21], text to image [25], or in fact a mixture of different mediums e.g . video and audio to text [3].”
Squeeze-and-Excitation Networks (2017, background): “Recent work has shown its applicability to tasks such as image captioning [4, 48] and lip reading [7].”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “Futhermore, multi-modal learning has been used for several applications such as lip reading [4, 20, 21], audio-visual source separation and localization [3, 7, 28, 29, 39, 77, 87, 91, 113, 114], speech recognition [2, 73, 84], efficient inference [30, 56], egocentric action recognition [51] and navi”
DeepFakes: a New Threat to Face Recognition? Assessment and Detection (2018, methodology): “PCA is applied to the joint audio-visual features to reduce the dimensionality of the blocks of features and long short-term memory (LSTM) [25] network is trained to separate tampered and non-tampered videos as proposed in [11].”
FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals (2019, background): “nly [49] 0.927 AUC Own, NA only [59] & [60], image only [51] 0.851 AUC Own, NA image only [51] 0.866 AUC FF [2] image only [52] 24.74% EER temp. vidTimit [61] audio only [52] 33.86% EER temp. AMI [62] audio only [52] 14.12% EER temp. grid [63] audio only [22] 99% Own, NA only 50 videos [50] 99% ”
A Comprehensive Deep Learning-Based Approach to Reduced Order Modeling of Nonlinear Time-Dependent Parametrized PDEs (2020, background): “Deep learning (DL) techniques have gained great attention in recent years in several areas like computer vision [49, 50], natural language processing [51, 52] and speech recognition [53, 54], due to their ability to discover pattern and extract features from massive datasets, in order to make predic”