Audio-Visual Instance Discrimination with Cross-Modal Agreement

By Pedro Morgado, Nuno Vasconcelos, Ishan Misra. 2020

314 citations recorded by the source corpus.

We present a self-supervised learning approach to learn audio-visual representations from video and audio. Our method uses contrastive learning for cross-modal discrimination of video from audio and vice-versa. We show that optimizing for cross-modal discrimination, rather than within-modal discrimination, is important to learn good representations from video and audio. With this simple but powerful insight, our method achieves highly competitive performance when finetuned on action recognition tasks. Furthermore, while recent work in contrastive learning defines positive and negative samples as individual instances, we generalize this definition by exploring cross-modal agreement. We group together multiple instances as positives by measuring their similarity in both the video and audio feature spaces. Cross-modal agreement creates better positive and negative sets, which allows us to calibrate visual similarities by seeking within-modal discrimination of positive instances, and achieve significant gains on downstream tasks.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Self-Supervised MultiModal Versatile Networks (2020, methodology): “Taking this idea into the self-supervised domain [8], multiple works use a pretext task of predicting whether visual and audio signals come from the same video [8, 9, 44, 58, 61, 72].”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, methodology): “There are many ways to construct v1 and v2: different augmentations of the same image [14, 5]; spatially adjacent image patches [36]; a video as v1 and its aligned audio as v2 for video representation learning [33, 8].”
What makes for good views for contrastive learning (2020, background): “For example: luminance and chrominance decomposition [53], randomly augmenting an image twice [58, 7, 4, 23, 60, 50, 65, 67], using different time-steps of videos [42, 66, 46, 22, 21], patches of the same image [27, 42, 26], multiple sensory data [39, 9, 44], or text and its context [37, 59, 35, 32]”
The Challenges of Continuous Self-Supervised Learning (2022, methodology): “coding [36, 54, 55], inpainting [59] and colorization [18, 34, 86], to methods that learn representations predictive of spatial context [19, 22, 53], temporal context [21, 47, 58, 61, 79, 80], or concurrent modalities like audio [4, 52, 57, 70], text [17, 23, 62] or speech [43, 44].”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “Multimodal self-supervised learning can be achieved by predicting whether a video has correspondence with an audio stream [4, 5, 68, 54], cross-modality clustering [3], and evolving losses [73].”
Support-set bottlenecks for video-text representation learning (2020, background): “Discriminative approaches have proven to be particularly successful for image (Chen et al., 2020c; He et al., 2020; Misra & van der Maaten, 2020; Wu et al., 2018) and video (Han et al., 2019; Morgado et al., 2020; Patrick et al., 2020) representation learning.”
Self-Supervised Pretraining of 3D Features on any Point-Cloud (2021, methodology): “These instance discrimination methods can be extended to multiple modalities [57, 62, 93].”
Divide and Contrast: Self-supervised Learning from Uncurated Data (2021, background): “Examples include different frames of video [69, 112, 81, 37, 31, 38], point clouds [100], multiple sensory data [65, 18, 74], text and its context [63, 101, 59, 51], or video and language [84, 62, 56].”
Contrastive Representation Learning: A Framework and Review (2020, background): “Similarly, Cross-modal Audio Visual Instance Discrimination (Cross-AVID) [72] jointly learn the general representation from video using corresponding image frames and audio segments.”
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction (2022, unknown): “…focuses on learning self-supervised audio-visual representations of general videos to solve high-level semantic tasks, e.g., action recognition and audio event detection (Arandjelovi ´ c & Zisserman, 2017; Bruno et al., 2018; Morgado et al., 2021; Chen et al., 2020; Lee et al., 2021).”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “…audio-visual correspondence in videos to formulate various self-supervised signals, which predict the cross-modal correspondence [253], [254], align the temporally corresponding representations [250], [255], or cluster their representations in a shared audio-visual latent space [172], [256].”
Seed the Views: Hierarchical Semantic Alignment for Contrastive Representation Learning (2022, background): “AVID [28] introduces multi-modality information to obtain reliable positive samples, where instances agree in both visual and audio feature space are considered as positives.”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “Morgado et al. [299] optimized visual similarity rather than simple cross-modal similarity using SS based on contrast learning with cross-modal audio and visual recognition.”
Multi-Format Contrastive Learning of Audio Representations (2021, background): “Meanwhile, recent works show that better representations can be learned by the proxy task of predicting whether the visual and audio signals come from the same video [22, 23, 24, 25, 26, 27, 28, 29, 30].”
CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding (2022, background): “[34] combines audio and video modalities to perform a cross-modal agreement which then achieves significant gains in action recognition and sound recognition tasks.”
Omnivore: A Single Model for Many Visual Modalities (2022, background): “For example, a range of tasks require training separate encoders for images and text [15, 30, 41, 57, 59], for video and audio [3, 4, 62, 63, 67, 71], or for video and optical flow [77].”
Parameter Efficient Multimodal Transformers for Video Representation Learning (2020, background): “, a visual clip matches with sound captured a few times steps before) (Kazakos et al., 2019; Morgado et al., 2020).”
A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning (2021, background): “With some of them using additional modalities of optical-flow [82,35], audio [3, 69, 2, 63] and text [80, 2] to transfer supervision from one modality to another.”