Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization

By Bruno Korbar, Du Tran, Lorenzo Torresani. 2018

514 citations recorded by the source corpus.

There is a natural correlation between the visual and auditive elements of a video. In this work we leverage this connection to learn general and effective models for both audio and video analysis from self-supervised temporal synchronization. We demonstrate that a calibrated curriculum learning scheme, a careful choice of negative examples, and the use of a contrastive loss are critical ingredients to obtain powerful multi-sensory representations from models optimized to discern temporal synchronization of audio-video pairs. Without further finetuning, the resulting audio features achieve performance superior or comparable to the state-of-the-art on established audio classification benchmarks (DCASE2014 and ESC-50). At the same time, our visual subnet provides a very effective initialization to improve the accuracy of video-based action recognition models: compared to learning from scratch, our self-supervised pretraining yields a remarkable gain of +19.9% in action recognition accuracy on UCF101 and a boost of +17.7% on HMDB51.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, background): “Prior works have leveraged ambient sound [52, 5, 4, 71, 34, 51, 55], dialogue [49], and narration [70, 2, 72, 73, 46, 45, 3], all of which of which serve as useful signals.”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “Taking this idea into the self-supervised domain [8], multiple works use a pretext task of predicting whether visual and audio signals come from the same video [8, 9, 44, 58, 61, 72].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “We outperform state-ofthe-art on UCF-101 and perform on par with AVTS [43] on HMDB-51.”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “For video data, these pretext tasks learn representations by ordering video frames [1, 18, 34, 41, 51, 79, 83], tracking [62, 77], or using cross-modal signals like audio [2, 3, 19, 36, 60, 61].”
Revisiting Self-Supervised Visual Representation Learning (2019, background): “Similarly, when learning representation from videos, one can either make use of the synchronized cross-modality stream of audio, video, and potentially subtitles [35, 39, 23, 44], or of the consistency in the temporal dimension [41].”
ActBERT: Learning Global-Local Video-Text Representations (2020, background): “The synchronized video-audio signal is also a good test-bed for cross-modal representation learning [3, 15].”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “Multimodal self-supervised learning can be achieved by predicting whether a video has correspondence with an audio stream [4, 5, 68, 54], cross-modality clustering [3], and evolving losses [73].”
Video Representation Learning by Dense Predictive Coding (2019, background): “Other modalities like audio signal can also benefits visual representation learning [18].”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, background): “…[28, 38], the context of images [25, 36], and the similarity of two patches from an image [6, 15, 33, 34, 48], and the other is the temporal based methods, which exploit temporal connection between frames such as the temporal order of frames [10, 26, 29] and cross-modal correspondence [23, 39].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “Some works propose proxy tasks to learn representations for downstream tasks such as classiﬁcation [9, 10, 12, 13, 52–54].”
Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, methodology): “A correct pair is an ‘in-sync’ video and audio and an incorrect pair can be constructed by using ‘out-of-sync’ audio [41] or audio from a different video [2].”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, background): “Examples for such pretext tasks are: colorizing grayscale images [100], image jigsaw puzzle [49], image inpainting [55], video frame order verification [45,92], video frame order recognition [32], video colorization [86], video future prediction [85,70], audio-visual correspondence [30,4], speech-vi”
Spatiotemporal Contrastive Video Representation Learning (2020, background): “Furthermore, videos can often supply multimodal signals for cross-modality self-supervision, such as geometric cues [19], speech or language [60, 59, 45], audio [40, 3, 51, 4], optical flow [30] or combinations of multiple modalities [2] and tasks [52].”
Multi-Modal Domain Adaptation for Fine-Grained Action Recognition (2019, background): “Self-supervision methods learn representations from the temporal [13, 59] and multi-modal structure of video [1, 25], leveraging pretraining on a large corpus of unlabelled videos.”
Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer (2021, background): “It exploits the image-audio alignment inherent in videos and learns audio and image / video representations jointly (Korbar et al., 2018; Wang et al., 2021; Nagrani et al., 2021).”
Evolving Losses for Unsupervised Video Representation Learning (2020, methodology): “Learning representations taking advantage of audio and video features has been explored by predicting if an audio clip is from a video [2] or if audio and video are temporally aligned [29, 7, 22, 3].”
Deep Audio-visual Learning: A Survey (2020, methodology): “To solve this task, Arandjelovic and Zisserman[16] proposed an L3-Net that detected whether the semantics in visual and audio fields were consistent.”
Hard Negative Mixing for Contrastive Learning (2020, background): “A number of works exploit the audio-visual nature of video [37, 1, 51, 14, 3] to learn visual representation, e.g . via learning intra-modality synchronization.”
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting (2021, methodology): “Existing methods for images [11, 23, 18] or videos [34] are designed for pixel perfect renderings of scenes or objects, and as such are not suited for sparse black and white handwritten images.”