Self-supervised audio representation learning for mobile devices

By Marco Tagliasacchi, Beat Gfeller, Felix de Chaumont Quitry, Dominik Roblek. 2019

48 citations recorded by the source corpus.

We explore self-supervised models that can be potentially deployed on mobile devices to learn general purpose audio representations. Specifically, we propose methods that exploit the temporal context in the spectrogram domain. One method estimates the temporal gap between two short audio segments extracted at random from the same audio clip. The other methods are inspired by Word2Vec, a popular technique used to learn word embeddings, and aim at reconstructing a temporal spectrogram slice from past and future slices or, alternatively, at reconstructing the context of surrounding slices from the current slice. We focus our evaluation on small encoder architectures, which can be potentially run on mobile devices during both inference (re-using a common learned representation across multiple downstream tasks) and training (capturing the true data distribution without compromising users' privacy when combined with federated learning). We evaluate the quality of the embeddings produced by the self-supervised learning models, and show that they can be re-used for a variety of downstream tasks, and for some tasks even approach the performance of fully supervised models of similar size.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “We train a convolutional network for speaker classification inspired by [58], which operates on the log-mel spectrogram of the inputs (25 ms window length, 10 ms hop length, 64 mel bins), cropped to 1 second.”
Contrastive Learning of General-Purpose Audio Representations (2020, methodology): “Despite recent progress, most work on learning representations of audio focuses on speech tasks [17, 18, 19] (with the exception of [6, 16]) and ignores other audio tasks such as acoustic scene detection or animal vocalizations.”
Self-Supervised Speech Representation Learning: A Review (2022, background): “[116], phase information [117], and the temporal gap between two segments [115], [116].”
Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision (2019, background): “For sound events, temporal proximity, temporal lag prediction, context reconstruction, and sound mixing have been demonstrated to be effective [11, 16].”
This paper is included in the Proceedings of the 2022 USENIX Annual Technical Conference. (unknown): “For example, training (fine-tuning) a small NN model Audio2Vec [15] can take about ten iterations, and losing three iterations for dynamic profiling leads to more than 20% increase in training time.”