Lipreading Using Temporal Convolutional Networks

By Brais Martínez, Pingchuan Ma, Stavros Petridis, Maja Pantic. IEEE International Conference on Acoustics, Speech, and Signal Processing, 2020

307 citations recorded by the source corpus.

Lip-reading has attracted a lot of research attention lately thanks to advances in deep learning. The current state-of-the-art model for recognition of isolated words in-the-wild consists of a residual network and Bidirectional Gated Recurrent Unit (BGRU) layers. In this work, we address the limitations of this model and we propose changes which further improve its performance. Firstly, the BGRU layers are replaced with Temporal Convolutional Networks (TCN). Secondly, we greatly simplify the training procedure, which allows us to train the model in one single stage. Thirdly, we show that the current state-of-the-art methodology produces models that do not generalize well to variations on the sequence length, and we address this issue by proposing a variable-length augmentation. We present results on the largest publicly-available datasets for isolated word recognition in English and Mandarin, LRW and LRW1000, respectively. Our proposed model1 results in an absolute improvement of 1.2% and 3.2%, respectively, in these datasets which is the new state-of-the-art performance.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/d8da125d2511d037df036c0f0da7c57135e24409

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection (2020, background): “net is a multi-scale temporal convolutional network (MSTCN) [40], combining short- and long-term temporal information at every layer by concatenating outputs of multiple branches, each with a different temporal receptive field.”
Detecting Deepfakes with Self-Blended Images (2022, background): “(3) LipForensics [27] detects temporal inconsistencies of mouth movements using a pretrained lipreading model [46].”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, methodology): “Following the state-of-the-art in lip reading [46, 44], the lip motion analysis network takes N mouth regions of interest (ROIs)2 as input and it consists of a 3D convolutional layer followed by a ShuffleNet v2 [43] network to extract a time-indexed sequence of feature vectors.”
End-To-End Audio-Visual Speech Recognition with Conformers (2021, methodology): “Data augmentation Following [20, 28], random cropping with a size of 88 × 88 and horizontal ﬂipping with a probability of 0.5 are performed for each image sequence.”
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction (2022, unknown): “The image encoder is a modified ResNet-18, which has been used in prior work (Ma et al., 2021b; Martinez et al., 2020; Stafylakis & Tzimiropoulos, 2017). in Makino et al. (2019).”