Multi-Format Contrastive Learning of Audio Representations

By Luyu Wang, Aäron van den Oord. 2021

64 citations recorded by the source corpus.

Recent advances suggest the advantage of multi-modal training in comparison with single-modal methods. In contrast to this view, in our work we find that similar gain can be obtained from training with different formats of a single modality. In particular, we investigate the use of the contrastive learning framework to learn audio representations by maximizing the agreement between the raw audio and its spectral representation. We find a significant gain using this multi-format strategy against the single-format counterparts. Moreover, on the downstream AudioSet and ESC-50 classification task, our audio-only approach achieves new state-of-the-art results with a mean average precision of 0.376 and an accuracy of 90.5%, respectively.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition (2021, neutral): “Previous SoTA Multi-format [103] AudioSet A 0.”
Self-Supervised Learning from Automatically Separated Sound Scenes (2021, methodology): “For general-purpose audio, including a variety of environmental sounds beyond speech, the majority of works are based on contrastive learning [14, 15, 16, 17, 18, 19], where a representation is learned by comparing pairs of examples selected by some semantically-correlated notion of similarity [20].”
Multimodal Self-Supervised Learning of General Audio Representations (2021, unknown): “Similar to [16], we ﬁnd that the log-mel networks outperform their waveform counterparts under all circum-stances.”
OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation (2021, background): “In audio and speech processing, pre-training has focused on emotion recognition [24], speaker identification [32], phoneme discrimination [39, 27], transferring ASR representations from one language to another [17], unsupervised representations learning for speech [35], audio representation learning”