Multimodal Self-Supervised Learning of General Audio Representations

By Luyu Wang, Pauline Luc, Adrià Recasens, Jean-Baptiste Alayrac, Aäron van den Oord. 2021

46 citations recorded by the source corpus.

We present a multimodal framework to learn general audio representations from videos. Existing contrastive audio representation learning methods mainly focus on using the audio modality alone during training. In this work, we show that additional information contained in video can be utilized to greatly improve the learned features. First, we demonstrate that our contrastive framework does not require high resolution images to learn good audio features. This allows us to scale up the training batch size, while keeping the computational load incurred by the additional video modality to a reasonable level. Second, we use augmentations that mix together different samples. We show that this is effective to make the proxy task harder, which leads to substantial performance improvements when increasing the batch size. As a result, our audio model achieves a state-of-the-art of 42.4 mAP on the AudioSet classification downstream task, closing the gap between supervised and self-supervised methods trained on the same dataset. Moreover, we show that our method is advantageous on a broad range of non-semantic audio tasks, including speaker identification, keyword spotting, language identification, and music instrument classification.

Subfields: Artificial Intelligence, Multimodal Learning, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yejin Choi
2. Mohit Bansal
3. Jianfeng Gao
4. Ali Farhadi
5. Luke S. Zettlemoyer
6. Percy Liang
7. Hannaneh Hajishirzi
8. Rowan Zellers

In Multimodal Learning

1. Ali Farhadi
2. Rowan Zellers
3. Yejin Choi
4. Jiasen Lu
5. Jack Hessel
6. Youngjae Yu
7. Ximing Lu
8. Yoav Artzi

In Representation & Self-Supervised Learning

1. Ali Farhadi
2. Kyu J. Han
3. Andreas R. Dengel
4. Jing Pan
5. Yejin Choi
6. Felix Wu
7. Rowan Zellers
8. Jack Hessel

In Speech & Audio

1. Kyu J. Han
2. Jing Pan
3. Felix Wu
4. Yoav Artzi
5. Kilian Q. Weinberger
6. Kwangyoun Kim

In Signal Processing

1. Andrey Guzhov
2. Ali Farhadi
3. Kyu J. Han
4. Andreas R. Dengel
5. Jing Pan
6. Yejin Choi
7. Felix Wu
8. Rowan Zellers

Citations in the corpus

• MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “Recent work has explored this paradigm for matching video frames with their transcripts [121], with their audio signal [96, 114], or both [3, 2]; these works likewise perform well on single-modality tasks like audio classification and activity recognition.”
• Performance-Efficiency Trade-Offs in Unsupervised Pre-Training for Speech Recognition (2021, methodology): “SEW-D outperforms W2V2 in most pre-training settings, when experimenting with LibriSpeech (Panayotov et al., 2015), Ted-lium 3 (Hernandez et al., 2018), VoxPopuli (Wang et al., 2021a), and Switchboard (Godfrey and Holliman, 1993) datasets.”
• Audioclip: Extending Clip to Image, Text and Audio (2021, methodology): “The joint use of several modalities occurred first in video-related [14,34,6] tasks and was adapted to the sound classification task later [13,31].”
• Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer (2021, background): “It exploits the image-audio alignment inherent in videos and learns audio and image / video representations jointly (Korbar et al., 2018; Wang et al., 2021; Nagrani et al., 2021).”