Unsupervised Learning of Semantic Audio Representations

By Aren Jansen, Manoj Plakal, R. Pandya, D. Ellis, Shawn Hershey, Jiayang Liu, R. Channing Moore, Rif A. Saurous. 2017

137 citations recorded by the source corpus.

Even in the absence of any explicit semantic annotation, vast collections of audio recordings provide valuable information for learning the categorical structure of sounds. We consider several class-agnostic semantic constraints that apply to unlabeled nonspeech audio: (i) noise and translations in time do not change the underlying sound category, (ii) a mixture of two sound events inherits the categories of the constituents, and (iii) the categories of events in close temporal proximity are likely to be the same or related. Without labels to ground them, these constraints are incompatible with classification loss functions. However, they may still be leveraged to identify geometric inequalities needed for triplet loss-based training of convolutional neural networks. The result is low-dimensional embeddings of the input spectrograms that recover 41% and 84% of the performance of their fully-supervised counterparts when applied to downstream query-by-example sound retrieval and sound event classification tasks, respectively. Moreover, in limited-supervision settings, our unsupervised embeddings double the state-of-the-art classification performance.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition (2021, background): “FOLLOWING STANDARD PRACTICE FOR EVALUATING UNSUPERVISED REPRESENTATIONS [102], THE REPRESENTATION IS HELD FIXED AND THE ENTIRE AUDIOSET TRAIN SET IS USED TO TRAIN AN MLP CLASSIFIER WITH A SINGLE 512-UNIT HIDDEN LAYER.”
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “For each ZeroSpeech testing waveform, we randomly sampled an AudioSet waveform of the same duration and performed linear mixing with a signal-to-noise ratio (SNR) selected randomly within a specified range.”
FSD50K: An Open Dataset of Human-Labeled Sound Events (2020, methodology): “FSD50K is an open and stable dataset aimed at complementing AudioSet in order to foster reproducible large-vocabulary SER research.”
Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks (2019, background): “Some attempts have also been done to extend self-supervised learning to different modalities [12,13] or to audio representations only [14–17].”
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input (2018, background): “Semi-hard negative mining (Jansen et al. 2018) is a variant of hard negative mining in which the impostors are constrained to be less similar to the anchor than its paired sample (Fig.”
Grounding Spoken Words in Unlabeled Video (2019, background): “The second loss term employs semi-hard negative mining [11].”
Contrastive Learning of General-Purpose Audio Representations (2020, methodology): “This approach learns a representation from pairs of similar inputs from unlabeled data, exploiting e.g. temporal consistency [5, 4, 6] or data augmentation [7] and trains a model to recognize similar elements among negative distractors.”
Transfer Learning from Audio-Visual Grounding to Speech Recognition (2019, methodology): “Following [25], imposter instances are drawn using a mixture of uniform sampling and within-batch semi-hard negative mining [24].”
Towards Visually Grounded Sub-word Speech Unit Discovery (2019, methodology): “First, we employ within-batch semi-hard negative mining [24, 25].”
Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision (2019, methodology): “Our prior work in temporal proximity-based metric learning [11] was a direct attempt to leverage coincidence for audio representation learning.”
Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion (2019, background): “Embedding audio signals into latent representations has been a well studied practice [4][5][6][7][8][9][10].”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, background): “Contrastive learning has also been explored for global speech representation learning (Milde and Biemann 2018; Jati and Georgiou 2019; Jansen et al. 2018).”
Unspeech: Unsupervised Speech Context Embeddings (2018, background): “[13] encode the notion that (non-speech) sounds occurring in context are more related using a triplet loss function.”
Multi-Format Contrastive Learning of Audio Representations (2021, methodology): “We evaluate the representations in the downstream task of training shallow fully connected audio classifiers following the same setup as in [20, 26], where a 1-hidden-layer MLP with 512 units is used and the parameters in the pretrained network are fixed.”
Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings (2019, background): “More recent approaches successfully leverage triplet learning [28] under contextual constraints, such as the temporal proximity of audio samples in video streams [12].”
Towards Learning a Universal Non-Semantic Representation of Speech (2020, methodology): “Following [18, 5], we (i) take as input log mel spectrogram context windows with F = 64 mel bands and T = 96 frames representing 0.”
A Deep Residual Network for Large-Scale Acoustic Scene Analysis (2019, methodology): “The ResNet architecture has been used in audio tasks such as speaker spoof detection [25] and unsupervised audio representation learning [26].”
Unsupervised Scalable Representation Learning for Multivariate Time Series (2019, methodology): “, 2019), but have not found much use for time series apart from audio (Bredin, 2017; Lu et al., 2017; Jansen et al., 2018), and never, to our knowledge, in a fully unsupervised setting, as existing works assume the existence of class labels or annotations in the training data.”
Unsupervised Contrastive Learning of Sound Event Representations (2020, methodology): “A similar approach is used in [7] to create examples for triplet loss-based training.”