Multi-Task Self-Supervised Learning for Robust Speech Recognition

By Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, João Monteiro, Jan "Yenda" Trmal, Yoshua Bengio. 2020

310 citations recorded by the source corpus.

Despite the growing interest in unsupervised learning, extracting meaningful knowledge from unlabelled audio remains an open challenge. To take a step in this direction, we recently proposed a problem-agnostic speech encoder (PASE), that combines a convolutional encoder followed by multiple neural networks, called workers, tasked to solve self-supervised problems (i.e., ones that do not require manual annotations as ground truth). PASE was shown to capture relevant speech information, including speaker voice-print and phonemes. This paper proposes PASE+, an improved version of PASE for robust speech recognition in noisy and reverberant environments. To this end, we employ an online speech distortion module, that contaminates the input signals with a variety of random disturbances. We then propose a revised encoder that better learns short- and long-term speech dynamics with an efficient combination of recurrent and convolutional networks. Finally, we refine the set of workers used in self-supervision to encourage better cooperation.Results on TIMIT, DIRHA and CHiME-5 show that PASE+ significantly outperforms both the previous version of PASE as well as common acoustic features. Interestingly, PASE+ learns transferable representations suitable for highly mismatched acoustic conditions.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, background): “Other recent work on pre-training which evaluates on TIMIT includes [47] who solve multiple tasks to learn good representations of speech.”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, methodology): “Table 5 shows that our best model outperforms previous work, improving the best known prior work [42] by 1% absolute, a relative error reduction of 15%.”
Unispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training (2021, methodology): “Self-supervised learning methods are categorized into discriminative methods [1, 2, 3, 4, 5, 6, 7], generative methods [8, 9, 10, 11, 12, 13], and multi-task learning methods [14].”
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, methodology): “learning is adopted in PASE [42] and PASE+ [43].”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “, 2020b), derived signal processing features (Pascual et al., 2019; Ravanelli et al., 2020), and binary labels of whether a candidate is the target unseen spectrogram (van den Oord et al.”
SUPERB: Speech processing Universal PERformance Benchmark (2021, methodology): “…have investigated these SSL models’ capabilities on tasks including phoneme classiﬁcation [11, 7], speaker identiﬁcation [7, 8], speaker ver-iﬁcation [17], emotion recognition [15], ASR [9, 12, 10, 16], speech translation [7], spoken language understanding [18], voice conversion [19] and TTS [20].”
Data Augmenting Contrastive Learning of Speech Representations in the Time Domain (2020, neutral): “Our work is close to [18], which applies data augmentation techniques to representation learning (autoencoders).”
Vector-Quantized Autoregressive Predictive Coding (2020, background): “The resulting representations, obtained without requiring any additional labeled data, are able to outperform the surface features across downstream tasks such as speech recognition, speech translation, speaker identiﬁcation, and emotion recognition [14, 15].”
Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations (2021, methodology): “Another popular self-supervised learning method for speech representation is to train a neural network by targeting multiple self-supervision tasks [34, 37].”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “PASE and PASE+ [69], [113] use multiple targets, including the waveform, log power spectrum, mel”
i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning (2021, methodology): “…be understood as the original data, data augmentations are carefully designed using the domain knowledge about images (DeVries & Taylor, 2017b; Cubuk et al., 2019a;b; Zhong et al., 2020), speech (Amodei et al., 2016; Park et al., 2019), or natural languages (Zhang et al., 2015; Wei & Zou, 2019).”
Contrastive Learning Inverts the Data Generating Process (2021, methodology): “…representations for visual and sequential data (Logeswaran & Lee, 2018; Wu et al., 2018; Oord et al., 2018; Hénaff, 2020; Tian et al., 2019; Hjelm et al., 2019; Bachman et al., 2019; He et al., 2020a; Chen et al., 2020a; Schneider et al., 2019; Baevski et al., 2020a;b; Ravanelli et al., 2020).”
Towards Unsupervised Learning of Speech Features in the Wild (2021, background): “In the low resource setting, features are viewed as pretraining and are evaluated as their ability to transfer to some downstream task like phone or word recognition [4, 7, 31, 32, 33].”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, methodology): “…2020; Liu, Li, and Lee 2021; Ling and Liu 2020; Hsu et al. 2021a; Chen et al. 2021), using a contrastive objective (v. d. Oord, Li, and Vinyals 2018; Schneider et al. 2019; Baevski, Schneider, and Auli 2020; Baevski et al. 2020), or multi-task learning (Pas-cual et al. 2019; Ravanelli et al. 2020).”
SSAST: Self-Supervised Audio Spectrogram Transformer (2021, methodology): “Finally, we pre-train the model with joint discriminative and generative objectives, which is also novel in the audio and speech domain and only has been explored in (Pascual et al. 2019; Jiang et al. 2020; Ravanelli et al. 2020).”
Attention Is All You Need In Speech Separation (2020, unknown): “We also randomly drop out some frequency channels by convolving the waveform with random band-pass filters as done in [25].”