Towards Unsupervised Learning of Speech Features in the Wild

By Morgane Rivière, Emmanuel Dupoux. 2021

21 citations recorded by the source corpus.

Recent work on unsupervised contrastive learning of speech representation has shown promising results, but so far has mostly been applied to clean, curated speech datasets. Can it also be used with unprepared audio data "in the wild"? Here, we explore three potential problems in this setting: (i) presence of non-speech data, (ii) noisy or low quality speech data, and (iii) imbalance in speaker distribution. We show that on the Libri-light train set, which is itself a relatively clean speech-only dataset, these problems combined can already have a performance cost of up to 30% relative for the ABX score. We show that the first two problems can be alleviated by data filtering, with voice activity detection selecting speech segments, while perplexity of a model trained with clean data helping to discard entire files. We show that the third problem can be alleviated by learning a speaker embedding in the predictive branch of the model. We show that these techniques build more robust speech features that can be transferred to an ASR task in the low resource setting.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, methodology): “We train CPCbased models (Riviere and Dupoux, 2020) on 500hour English and 500-hour French unlabeled data from VoxPopuli, respectively.”
AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “While previous works [14], [45] use the 6k-hour clean subset [55] of Libri-Light for training the language model,”
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations (2021, methodology): “For CPC, we used the model from [44], which was trained on a “clean” 6k hour sub-sample of the LibriLight dataset [45, 44].”
On Generative Spoken Language Modeling from Raw Audio (2021, positive): “This model was trained on a “clean” 6k hour sub-sample of the LibriLight dataset (Kahn et al., 2020; Rivière and Dupoux, 2020).”
Generative Spoken Dialogue Language Modeling (2022, methodology): “were selected to be of high intelligibility using the average perplexity of a phone recognizer trained on the clean Librispeech 100h training subset (Rivière and Dupoux, 2021).”
Text-Free Prosody-Aware Generative Spoken Language Modeling (2021, methodology): “In our experiments, we train MS-TLM models on two English datasets: LibriSpeech (Panayotov et al., 2015) and a 6K-hour subset (Rivière and Dupoux, 2020) of Libri-Light (Kahn et al., 2020) which we refer to as LL-6K.”
textless-lib: a Library for Textless Spoken Language Processing (2022, methodology): “, 2021) trained on LibriSpeech 960h dataset, with a framerate of 50 Hz; (ii) Contrastive Predictive Coding (CPC) model (Rivière and Dupoux, 2020; Oord et al., 2018) trained on the 6K hours subset from LibriLight (Kahn et al.”