Perfect Match: Improved Cross-modal Embeddings for Audio-visual Synchronisation

By Soo-Whan Chung, Joon Son Chung, Hong-Goo Kang. 2018

133 citations recorded by the source corpus.

This paper proposes a new strategy for learning powerful cross-modal embeddings for audio-to-video synchronisation. Here, we set up the problem as one of cross-modal retrieval, where the objective is to find the most relevant audio segment given a short video clip. The method builds on the recent advances in learning representations from cross-modal self-supervision. The main contributions of this paper are as follows: (1) we propose a new learning strategy where the embeddings are learnt via a multi-way matching problem, as opposed to a binary classification (matching or non-matching) problem as proposed by recent papers; (2) we demonstrate that performance of this method far exceeds the existing baselines on the synchronisation task; (3) we use the learnt embeddings for visual speech recognition in self-supervision, and show that the performance matches the representations learnt end-to-end in a fully-supervised manner.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Signal Processing

Citations in the corpus

Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, methodology): “There are many ways to construct v1 and v2: different augmentations of the same image [14, 5]; spatially adjacent image patches [36]; a video as v1 and its aligned audio as v2 for video representation learning [33, 8].”
What makes for good views for contrastive learning (2020, background): “For example: luminance and chrominance decomposition [53], randomly augmenting an image twice [58, 7, 4, 23, 60, 50, 65, 67], using different time-steps of videos [42, 66, 46, 22, 21], patches of the same image [27, 42, 26], multiple sensory data [39, 9, 44], or text and its context [37, 59, 35, 32]”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, background): “Leveraging this link, cross-modal learning methods explore a range of interesting tasks: face reconstruction from audio [54], talking face generation [82], emotion recognition [5], speaker diarization [13], speech recognition [16], and speaker identification [50, 49, 39, 74, 17].”
Is Someone Speaking?: Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection (2021, unknown): “In audio-visual synchronization studies, by using convolutional neural network (CNN) [18, 28], LSTM [39] or attention model [11], the longer video utterance are used, the more representative features can be extracted, which”
Divide and Contrast: Self-supervised Learning from Uncurated Data (2021, background): “Examples include different frames of video [69, 112, 81, 37, 31, 38], point clouds [100], multiple sensory data [65, 18, 74], text and its context [63, 101, 59, 51], or video and language [84, 62, 56].”
Self-Supervised Learning of Audio-Visual Objects from Video (2020, methodology): “While this is a problem that is typically solved using supervised face detection [13, 15], we instead tackle it with our learned model.”
Deep Audio-visual Learning: A Survey (2020, background): “To further enhance the robustness of the network, Chung et al. [46] proposed an improved two-stream training method that increased the number of negative samples to improve the error-tolerance rate of the network.”
Spot the conversation: speaker diarisation in the wild (2020, methodology): “The first method uses a variant [30] of SyncNet [31], which is a two-stream CNN that determines the active speaker by estimating the correlation between the audio track and the mouth motion of the video.”