Data Augmenting Contrastive Learning of Speech Representations in the Time Domain

By Evgeny Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Pierre-Emmanuel Mazar'e, Pierre-Emmanuel Mazaré, Matthijs Douze, Emmanuel Dupoux. 2020

129 citations recorded by the source corpus.

Contrastive Predictive Coding (CPC), based on predicting future segments of speech from past segments is emerging as a powerful algorithm for representation learning of speech signal. However, it still under-performs compared to other methods on unsupervised evaluation benchmarks. Here, we intro-duce WavAugment, a time-domain data augmentation library which we adapt and optimize for the specificities of CPC (raw waveform input, contrastive loss, past versus future structure). We find that applying augmentation only to the segments from which the CPC prediction is performed yields better results than applying it also to future segments from which the samples (both positive and negative) of the contrastive loss are drawn. After selecting the best combination of pitch modification, additive noise and reverberation on unsupervised metrics on LibriSpeech (with a gain of 18-22% relative on the ABX score), we apply this combination without any change to three new datasets in the Zero Resource Speech Benchmark 2017 and beat the state-of-the-art using out-of-domain training data. Finally, we show that the data-augmented pretrained features improve a downstream phone recognition task in the Libri-light semi-supervised setting (10 min, 1 h or 10 h of labelled data) reducing the PER by 15% relative.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, background): “Examples of pretext tasks for self-supervised speech representation learning include distinguishing near-by features from temporally distant ones [1]–[3], next-step prediction of audio features [4], masked prediction of audio features given unmasked context [5], [6].”
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, methodology): “We augment the data with time dropout, pitch modification and reverberation (Kharitonov et al., 2020) during model training.”
AudioLM: A Language Modeling Approach to Audio Generation (2022, background): “for example, two segments that are close temporally [28]–[30] or two augmented views of the same sequence [31].”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “Unsupervised speech representation learning aims to distill features useful for downstream tasks, such as phone discrimination (Kharitonov et al., 2020; Schneider et al., 2019) and semantic prediction (Lai et al.”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, background): “Most work applied pre-trained speech models to only inference problems such as ASR (Baevski et al., 2019) or phone discrimination (Kharitonov et al., 2020), with a notable exception being Tjandra et al. (2019a), which focuses on text-free machine translation.”
NoiseVC: Towards High Quality Zero-Shot Voice Conversion (2021, background): “On the other side, works [23, 24, 25] prove that Contrastive Predictive Coding (CPC) can largely enhance the phoneme representation learning from speech audios.”
textless-lib: a Library for Textless Spoken Language Processing (2022, methodology): “…across different devices; (ii) in cases where on-the-fly preprocessing is not required (e.g., there is no randomized data augmentation (Kharitonov et al., 2021b)), an alternative is to preprocess the entire dataset in advance. textless-lib provides a tool for preprocessing arbitrary…”
Textless Speech Emotion Conversion using Decomposed and Discrete Representations (2021, methodology): “Unsupervised speech representation learning aims to distill features useful for downstream tasks, such as phone discrimination (Kharitonov et al., 2021b; Schneider et al., 2019) and semantic prediction (Lai et al.”
Multi-Format Contrastive Learning of Audio Representations (2021, methodology): “We consider different types of augmentations for raw audio and spectrograms [20, 32, 33].”
Joint Masked CPC And CTC Training For ASR (2020, background): “While these methods [5, 6] have achieved impressive results on low-resource speech datasets, their goal is to learn speech representations that are useful for multiple speech-related tasks.”
Towards Unsupervised Learning of Speech Features in the Wild (2021, methodology): “Besides, in order to take full advantage of the labelled data, we perform some pitch augmentation using the WavAugment library as described in [40].”
Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training? (2021, background): “Prediction-based self-supervised learning has gathered increasing interests recently, where a model is tasked to predict the content of the unseen regions [29, 30, 31, 32, 33, 34, 35] or to contrast the target unseen frame with randomly sampled ones [5, 36, 6, 11].”
Unsupervised Contrastive Learning of Sound Event Representations (2020, methodology): “After achieving state-of-the-art in image recognition [9], this approach has been successfully applied for speech recognition [14, 15].”
End-To-End Audio-Visual Speech Recognition with Conformers (2021, methodology): “Similarly to [13], 2 sets of consecutive audio samples with a maximum length of 0.4 seconds are set to zero and 2 sets of consecutive frequency bands with a maximum width of 150 Hz are rejected.”
Self-supervised Context-aware Style Representation for Expressive Speech Synthesis (2022, unknown): “Besides, contrastive learning is also widely used in NLP and speech areas [13, 14, 15, 16].”
Contrastive Learning of General-Purpose Audio Representations (2020, methodology): “We generate similar pairs by simply sampling segments from the same audio clip, which avoids exploring augmentation strategies entirely unlike S IM CLR, M O C O , CURL and others [22].”
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction (2022, unknown): “Recent successful representation learning frameworks for speech (e.g., APC (Chung et al., 2019), CPC (Oord et al., 2018; Kharitonov et al., 2021), wav2vec 2.0 (Baevski et al., 2020; Hsu et al., 2021b), De-CoAR2.”