Contrastive Learning of General-Purpose Audio Representations

By Aaqib Saeed, David Grangier, Neil Zeghidour. 2020

333 citations recorded by the source corpus.

We introduce COLA, a self-supervised pre-training approach for learning a general-purpose representation of audio. Our approach is based on contrastive learning: it learns a representation which assigns high similarity to audio segments extracted from the same recording while assigning lower similarity to segments from different recordings. We build on top of recent advances in contrastive learning for computer vision and reinforcement learning to design a lightweight, easy-to-implement self-supervised model of audio. We pre-train embeddings on the large-scale Audioset database and transfer these representations to 9 diverse classification tasks, including speech, music, animal sounds, and acoustic scenes. We show that despite its simplicity, our method significantly outperforms previous self-supervised systems. We furthermore conduct ablation studies to identify key design choices and release a library1 to pre-train and fine-tune COLA models.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, methodology): “This training scheme has been successfully employed in different fields such as image and video classification [25], [53], [54], as well as natural language processing [55], [56] and audio representation learning [57], [58], [59].”
AudioLM: A Language Modeling Approach to Audio Generation (2022, background): “Positive pairs can be, for example, two segments that are close temporally [28]–[30] or two augmented views of the same sequence [31].”
Pengi: An Audio Language Model for Audio Tasks (2023, methodology): “In the domain of audio processing, Transfer Learning has facilitated the rise of Self-Supervised Learning and Zero-Shot Learning techniques [52, 53, 45, 24, 27, 26, 3, 17, 28, 25, 59, 60, 43, 13, 16, 18, 65, 10].”
Multimodal Self-Supervised Learning of General Audio Representations (2021, neutral): “COLA [14] Ours Sup.”
Self-Supervised Learning from Automatically Separated Sound Scenes (2021, methodology): “For general-purpose audio, including a variety of environmental sounds beyond speech, the majority of works are based on contrastive learning [14, 15, 16, 17, 18, 19], where a representation is learned by comparing pairs of examples selected by some semantically-correlated notion of similarity [20].”
MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training (2023, background): “Existing research work targeting general-purpose audio representations (Saeed et al., 2021; Borsos et al., 2022; Wang et al., 2023) has verified that self-supervised speech processing models can be extended beyond speech by adapting them to downstream entry-level music tasks, including generating mo”
Computational bioacoustics with deep learning: a review and roadmap (2021, background): “…Morﬁ and Stowell, 2018; LeBien et al., 2020); • self-supervised learning, which uses some aspect of the data itself as a substitute for supervised labelling (Saeed et al., 2021); • few-shot learning, in which a system is trained across multiple similar tasks, in such a way that for a new unseen…”
BEATs: Audio Pre-Training with Acoustic Tokenizers (2022, methodology): “The self-supervised audio pre-training methods typically learn the audio representations with the contrastive learning (Ravanelli & Bengio, 2018; Saeed et al., 2021; Fonseca et al., 2021; Al-Tahan & Mohsenzadeh, 2021; Wang & Oord, 2021) or reconstruction objective (Tagliasacchi et al.”
Performance-Efficiency Trade-Offs in Unsupervised Pre-Training for Speech Recognition (2021, background): “…interest in learning representations for other speech tasks (Synnaeve and Dupoux, 2016; Chung et al., 2018; Chuang et al., 2019; Song et al., 2019), music (Yang et al., 2021; Zhao and Guo, 2021), and general audio (Saeed et al., 2021; Gong et al., 2021; Niizumi et al., 2021; Wang et al., 2021c).”