Co-Training of Audio and Video Representations from Self-Co-Training of Audio and Video Representations from Self-Supervised Temporal Synchronization Supervised Temporal Synchronization

By Bruno Korbar, Du Tran, Lorenzo Torresani. 2018

Subfields: Multimodal Learning, Representation & Self-Supervised Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

In Representation & Self-Supervised Learning

In Speech & Audio

Citations in the corpus

Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning (2018, background): “…in computer vision has successfully used a very similar idea for “self-supervised” audio-visual feature extraction from a purely heuristic perspective (Arandjelovic and Zisserman, 2017; Korbar et al., 2018), and our theory may serve to elucidate the mathematical principles underlying such methods.”
Self-Supervised Learning of Audio-Visual Objects from Video (2020, methodology): “Recent efforts learn audio and video representations using self-supervised learning [13, 40, 46] with synchronization as the proxy task: the network has to predict whether video and audio are temporally aligned (or synthetically shifted).”
Deep Audio-visual Learning: A Survey (2020, background): “[13] Proposed the AVTS task with curriculum learning The sound source has to feature in the video; only one sound source Parekh et al.”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, background): “Apart from source separation, recent inspiring work integrates both audio and visual cues on an array of other tasks including self-supervised representation learning [57, 6, 8, 55, 41, 23], localizing sounds in video frames [7, 66, 71, 36], generating sounds from video [56, 84, 25, 47, 83], and act”
2.5D Visual Sound (2018, background): “Recent work explores self-supervision for visual [33, 2] and audio [4] feature learning, cross-modal representations [5], and audio-visual alignment [31, 23, 16].”
Learning to Separate Object Sounds by Watching Unlabeled Video (2018, background): “While a resurgence of research on cross-modal learning from images and audio also cap-italizes on synchronized audio-visual data for various tasks [3,4,5,47,49,59,60], they treat the audio as a single monolithic input, and thus cannot associate diﬀerent sounds to diﬀerent objects in the same video.”