Out of Time: Automated Lip Sync in the Wild

By Joon Son Chung, Andrew Zisserman. 2016

1,027 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

VoxCeleb2: Deep Speaker Recognition (2018, methodology): “This is done by using a multi-view adaptation [36] of ‘SyncNet’ [37, 38], a two-stream CNN which determines the active speaker by estimating the correlation between the audio track and the mouth motion of the video.”
VoxCeleb: A Large-Scale Speaker Identification Dataset (2017, methodology): “This is done by using ‘SyncNet’, a two-stream CNN described in [36] which estimates the correlation between the audio track and the mouth motion of the video.”
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, methodology): “Chung and Zisserman [22], the most closely related approach, train a two-stream network with an embedding loss.”
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (2018, methodology): “Inspired by similar ﬁndings in Chung et al [26], we discovered experimentally that more consistent and robust optimization can be obtained by minimizing the contrastive loss, which was originally introduced for training Siamese networks [27] on same-modality input pairs.”
Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, background): “Examples include the motion and appearance of a video [30, 80], depth and appearance [36, 91], luminance and chrominance of an image [80, 91], or as in our work sound and video [2, 4, 12, 63].”
Realistic Speech-Driven Facial Animation with GANs (2019, methodology): “istance will increase on either side of point where the minimum distance occurs. However, for uncorrelated audio and video the distance is expected to be stable. Based on this ﬂuctuation Chung et al. [12] further propose using the difference between the minimum and the median of the Euclidean distan”
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild (2018, methodology): “face-tracks, in which we have a known correspondence between the speech audio and the face depicted. Fortunately, this can be acquired, automatically and at scale using the recently developed SyncNet [18]. This method was used to generate the large-scale VoxCeleb dataset [50] for speaking face-track”
Learnable PINs: Cross-Modal Embeddings for Person Identity (2018, methodology): “The dataset contains over 100, 000 segmented speaking face-tracks obtained using SyncNet [13] from over 20k challenging videos.”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “Videos also contain useful multi-modal signals such as audio-visual synchronization [6, 20, 55, 77].”
Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset (2021, methodology): “The experiments are conducted on HDTF dataset with the metric of audio-visual synchronization[10] † .”
Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision (2019, background): “…(DeepCCA) [26], a deep learning generalization of linear CCA; (ii) Look, Listen, and Learn ( L 3 ) [12], which learns representations that can predict audio-visual frame correspondence in videos; and (iii) metric learning losses applied to shared multi-modal embedding spaces [27, 28, 29, 30].”
You said that? (2017, methodology): “This is similar to the conﬁguration used to learn audio embedding in [5].”
Lip Reading in Profile (2017, methodology): “As shown in [10], the visual stream of this network generates excellent features for the task of lip reading – on the LRW [9] and OuluVS2 [2] datasets, single-layer classifiers trained on the SyncNet features have outperformed networks trained end-to-end on the task.”
Self-Supervised Learning of Audio-Visual Objects from Video (2020, methodology): “Later, Chung and Zisserman [13] used synchronization cues to solve the active speaker detection problem.”
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (2021, methodology): “Extracting the shared information between visual and audio representations could lead to the speech content space by synchronizing both the modalities, which is also proven to be beneﬁcial for various downstream audio-visual tasks [17, 40, 73, 36, 25].”
Co-Training of Audio and Video Representations from Self-Co-Training of Audio and Video Representations from Self-Supervised Temporal Synchronization Supervised Temporal Synchronization (2018, methodology): “[11] in c).”
Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss (2019, methodology): “All of them are trained on LRW dataset while Chung et al. [3] require extra VGG-M network pretrained on VGG Face dataset [25] and Wilels et al. [35] need extra MFCC feature extractor pretrained by [5].”