Self-Supervised Learning by Cross-Modal Audio-Video Clustering

By Humam Alwassel, Dhruv Kumar Mahajan, Lorenzo Torresani, Bernard Ghanem, Du Tran. 2019

478 citations recorded by the source corpus.

Visual and audio modalities are highly correlated, yet they contain different information. Their strong correlation makes it possible to predict the semantics of one from the other with good accuracy. Their intrinsic differences make cross-modal prediction a potentially more rewarding pretext task for self-supervised learning of video and audio representations compared to within-modality learning. Based on this intuition, we propose Cross-Modal Deep Clustering (XDC), a novel self-supervised method that leverages unsupervised clustering in one modality (e.g., audio) as a supervisory signal for the other modality (e.g., video). This cross-modal supervision helps XDC utilize the semantic correlation and the differences between the two modalities. Our experiments show that XDC outperforms single-modality clustering and other multi-modal variants. XDC achieves state-of-the-art accuracy among self-supervised methods on multiple video and audio benchmarks. Most importantly, our video model pretrained on large-scale unlabeled data significantly outperforms the same model pretrained with full-supervision on ImageNet and Kinetics for action recognition on HMDB51 and UCF101. To the best of our knowledge, XDC is the first self-supervised learning method that outperforms large-scale fully-supervised pretraining for action recognition on the same architecture.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Are we done with ImageNet? (2020, neutral): “Similarly, successful representation learning techniques applied to ImageNet [9, 10, 11] have yielded corresponding gains elsewhere [12, 13, 14, 15].”
Learning Video Representations from Textual Web Supervision (2020, background): “Prior works have leveraged ambient sound [52, 5, 4, 71, 34, 51, 55], dialogue [49], and narration [70, 2, 72, 73, 46, 45, 3], all of which of which serve as useful signals.”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “3), the network ingests 2 seconds of audio for fair comparison to [7, 44], otherwise we use the same duration as the input video clip.”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “Multimodal self-supervised learning can be achieved by predicting whether a video has correspondence with an audio stream [4, 5, 68, 54], cross-modality clustering [3], and evolving losses [73].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “Some works propose proxy tasks to learn representations for downstream tasks such as classiﬁcation [9, 10, 12, 13, 52–54].”
Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, methodology): “Cross-AVID also outperforms the concurrently proposed XDC. Figure 3 shows the transfer performance as a function of the number of model parameters and FLOPs.”
Spatiotemporal Contrastive Video Representation Learning (2020, methodology): “Furthermore, videos can often supply multimodal signals for cross-modality self-supervision, such as geometric cues [19], speech or language [60, 59, 45], audio [40, 3, 51, 4], optical flow [30] or combinations of multiple modalities [2] and tasks [52].”
Multi-Format Contrastive Learning of Audio Representations (2021, background): “Meanwhile, recent works show that better representations can be learned by the proxy task of predicting whether the visual and audio signals come from the same video [22, 23, 24, 25, 26, 27, 28, 29, 30].”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “[367] leveraged unsupervised clustering in the audio/video modality as the supervisory signal for the video/audio modality, respectively.”
BridgeFormer: Bridging Video-text Retrieval with Multiple Choice Questions (2022, methodology): “…higher accuracy than some previous work that pre-train their model on datasets with considerably longer video time ( e.g. 14 × longer in XDC [3], 10 × longer in MIL-NCE [25] and VATT [1]), showing the effectiveness of our method in learning transferable video representations for action…”
A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning (2021, methodology): “The strongest previous approaches are using multi-modal input, Vision “V”, Audio “A”, Text “T”, to train a contrastive objective across modalities; XDC [3] performs DeepCluster [8] on (V+A), CVRL [71], GDT [68] and MMV [2] use an objective similar to SimCLR on (V), (V+A), and method pre-train backbo”
Divide and Contrast: Self-supervised Learning from Uncurated Data (2021, methodology): “Another line of methods generates pseudo labels by clustering features [8, 9, 46, 105, 1].”
Cross-Modal learning for Audio-Visual Video Parsing (2021, unknown): “There have been previous attempts toward self-supervised audio-visual pre-training such as [12, 13, 14].”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “…audio-visual correspondence in videos to formulate various self-supervised signals, which predict the cross-modal correspondence [253], [254], align the temporally corresponding representations [250], [255], or cluster their representations in a shared audio-visual latent space [172], [256].”
MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval (2022, background): “For example, XDC [3] pre-trains on IG-Kinetics that is 14 × longer than our pre-training dataset, but gets worse results.”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “After removing samples without audio, as in [2], we are left with around 210 K videos which we use for pretraining.”
Hard Negative Mixing for Contrastive Learning (2020, background): “A number of works exploit the audio-visual nature of video [37, 1, 51, 14, 3] to learn visual representation, e.g . via learning intra-modality synchronization.”
Parameter Efficient Multimodal Transformers for Video Representation Learning (2020, background): “Recent approaches leverage unlabeled videos from speciﬁc domains (Owens et al., 2016; Gao & Grauman, 2019; Zhao et al., 2018; Ephrat et al., 2018; Alwassel et al., 2019; Miech et al., 2020; Piergiovanni et al., 2020) and often demonstrate on audio-visual source separation, localization, and…”