Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks

By Santiago Pascual, Mirco Ravanelli, Joan Serrà, Antonio Bonafonte, Yoshua Bengio. 2019

259 citations recorded by the source corpus.

Learning good representations without supervision is still an open issue in machine learning, and is particularly challenging for speech signals, which are often characterized by long sequences with a complex hierarchical structure. Some recent works, however, have shown that it is possible to derive useful speech representations by employing a self-supervised encoder-discriminator approach. This paper proposes an improved self-supervised method, where a single neural encoder is followed by multiple workers that jointly solve different self-supervised tasks. The needed consensus across different tasks naturally imposes meaningful constraints to the encoder, contributing to discover general representations and to minimize the risk of learning superficial ones. Experiments show that the proposed approach can learn transferable, robust, and problem-agnostic features that carry on relevant information from the speech signal, such as speaker identity, phonemes, and even higher-level features such as emotional cues. In addition, a number of design choices make the encoder easily exportable, facilitating its direct usage or adaptation to different problems.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, background): “Other objectives do not belong to these categories, for example, [53].”
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (2021, background): “Self-supervised learning of speech representations has received a lot attention [1, 2, 3, 4, 5] and has been demonstrated to work well both in low- and high-resource labeled data settings for automatic speech recognition (ASR) [6].”
Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction (2020, background): “[6] learn unsupervised convolutional network-based representa-”
Multi-Task Self-Supervised Learning for Robust Speech Recognition (2020, positive): “In this section, we describe how PASE+ is pre-trained in a self-supervised manner, with a particular focus on the main improvements proposed on top of the original [15] (cf.”
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “Deep neural network models have been successfully used to learn powerful speech representations using weakly or unsupervised objectives (Thiolliere et al., 2015; Kamper et al., 2015; Hsu et al., 2017a;b; Hsu & Glass, 2018; Holzenberger et al., 2018; Milde & Biemann, 2018; van den Oord et al., 2018; ”
Improving Transformer-based Speech Recognition Using Unsupervised Pre-training (2019, background): “There are multiple concurrent approaches that generalize this approach and applied it in learning speaker representation [8], extracting speech representation [5, 9] and performing speech emotion recognition [6].”
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, methodology): “Multi-task learning is adopted in PASE [43] and PASE+ [44].”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “, 2020b), derived signal processing features (Pascual et al., 2019; Ravanelli et al., 2020), and binary labels of whether a candidate is the target unseen spectrogram (van den Oord et al.”
SUPERB: Speech processing Universal PERformance Benchmark (2021, background): “…have investigated these SSL models’ capabilities on tasks including phoneme classiﬁcation [11, 7], speaker identiﬁcation [7, 8], speaker ver-iﬁcation [17], emotion recognition [15], ASR [9, 12, 10, 16], speech translation [7], spoken language understanding [18], voice conversion [19] and TTS [20].”
Zero-Shot Multi-Speaker Text-To-Speech with State-Of-The-Art Neural Speaker Embeddings (2019, methodology): “Several recent studies [9, 13, 14, 15] have used this approach for speaker modeling in TTS, with [15] modeling both speaker and language characteristics.”
Generative Pre-Training for Speech with Autoregressive Predictive Coding (2019, methodology): “In some cases PASE is slightly better than T-APC10 (e.g., when only 1/8 or less of si284 is available), but is still worse than R-APC10.”
Vector-Quantized Autoregressive Predictive Coding (2020, background): “Recently, self-supervised learning—a paradigm that treats the input itself or modiﬁcations of the input as learning targets—has obtained promising results for learning such transformations [1, 2, 4, 5, 6, 7, 8, 9, 3, 10].”
Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations (2021, methodology): “Another popular self-supervised learning method for speech representation is to train a neural network by targeting multiple self-supervision tasks [34, 37].”
Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies (2020, background): “While speech representations can be defined by different transformations on the surface feature, recent researches [1, 2, 3, 4, 5, 6, 7, 8, 9] have shown great success by combining neural networks and self-supervised learning (where learning targets can be derived from the input itself).”
Contrastive Learning of General-Purpose Audio Representations (2020, unknown): “In contrast with generative encoder-decoder approaches [8, 9, 10, 11, 12], DPT is computationally efﬁcient as it avoids input reconstruction entirely.”
Investigating Self-Supervised Pre-Training for End-to-End Speech Translation (2020, background): “Finally, several self-supervised tasks can be jointly tackled to discover better speech representations [13].”
Self-supervised audio representation learning for mobile devices (2019, background): “Another interesting direction is merging representations learned by different self-supervised models, as recently proposed in [34] for the case of speech embeddings.”