Learning to lip read words by watching videos

By Joon Son Chung, Andrew Zisserman. 2018

96 citations recorded by the source corpus.

Abstract Our aim is to recognise the words being spoken by a talking face, given only the video but not the audio. Existing works in this area have focussed on trying to recognise a small number of utterances in controlled environments (e.g. digits and alphabets), partially due to the shortage of suitable datasets. We make three novel contributions: first, we develop a pipeline for fully automated data collection from TV broadcasts. With this we have generated a dataset with over a million word instances, spoken by over a thousand different people; second, we develop a two-stream convolutional neural network that learns a joint embedding between the sound and the mouth motions from unlabelled data. We apply this network to the tasks of audio-to-video synchronisation and active speaker detection; third, we train convolutional and recurrent networks that are able to effectively learn and recognize hundreds of words from this large-scale dataset. In lip reading and in speaker detection, we demonstrate results that exceed the current state-of-the-art on public benchmark datasets.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Signal Processing

Citations in the corpus

VoxCeleb2: Deep Speaker Recognition (2018, methodology): “This is done by using a multi-view adaptation [36] of ‘SyncNet’ [37, 38], a two-stream CNN which determines the active speaker by estimating the correlation between the audio track and the mouth motion of the video.”
Perfect Match: Improved Cross-modal Embeddings for Audio-visual Synchronisation (2018, methodology): “The ‘5’ refers to the receptive field of the feature extractor in the temporal dimension, in line with the naming convention of the networks in [15].”