Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings

By Jason Cramer, Ho-Hsiang Wu, Justin Salamon, Juan Pablo Bello. IEEE International Conference on Acoustics, Speech, and Signal Processing, 2019

365 citations recorded by the source corpus.

A considerable challenge in applying deep learning to audio classification is the scarcity of labeled data. An increasingly popular solution is to learn deep audio embeddings from large audio collections and use them to train shallow classifiers using small labeled datasets. Look, Listen, and Learn (L3-Net) is an embedding trained through self-supervised learning of audio-visual correspondence in videos as opposed to other embeddings requiring labeled data. This framework has the potential to produce powerful out-of-the-box embeddings for downstream audio classification tasks, but has a number of unexplained design choices that may impact the embeddings’ behavior. In this paper we investigate how L3-Net design choices impact the performance of downstream audio classifiers trained with these embeddings. We show that audio-informed choices of input representation are important, and that using sufficient data for training the embedding is key. Surprisingly, we find that matching the content for training the embedding to the downstream task is not beneficial. Finally, we show that our best variant of the L3-Net embedding outperforms both the VGGish and SoundNet embeddings, while having fewer parameters and being trained on less data. Our implementation of the L3-Net embedding model as well as pre-trained models are made freely available online.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/586afdbc1ada3a9e3e0bbc36cc334faca4d26f4a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representations (2020, methodology): “OpenL3 (Cramer et al., 2019) is an open source implementation of Look, Listen, and Learn (L3-Net) (Arandjelovic”
SONYC-UST-V2: An Urban Sound Tagging Dataset with Spatiotemporal Context (2020, methodology): “Audio content is given as OpenL3 [9] embeddings (with content type="env", input repr="mel256", and embedding size=512), using a window size and hop size of 1.”
Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision (2019, methodology): “We propose a general unimodal and cross-modal representation learning technique based on the proxy task of coincidence prediction, which uniﬁes recent work in audio-only [11] and audio-visual [12, 13] self-supervised learning.”
Self-supervised audio representation learning for mobile devices (2019, background): “Learning multimodal representations : Several papers have recently investigated learning audio representations exploiting the correlation with other modalities, e.g., text [8], images [32] and videos [31, 16, 1, 22, 10].”