Unispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training

By Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen, Shujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li and 1 more. 2021

138 citations recorded by the source corpus.

Self-supervised learning (SSL) is a long-standing goal for speech processing, since it utilizes large-scale unlabeled data and avoids extensive human labeling. Recent years have witnessed great successes in applying self-supervised learning in speech recognition, while limited exploration was attempted in applying SSL for modeling speaker characteristics. In this paper, we aim to improve the existing SSL framework for speaker representation learning. Two methods are introduced for enhancing the unsupervised speaker information extraction. First, we apply multi-task learning to the current SSL framework, where we integrate utterance-wise contrastive loss with the SSL objective function. Second, for better speaker discrimination, we propose an utterance mixing strategy for data augmentation, where additional overlapped utterances are created unsupervisely and incorporated during training. We integrate the proposed methods into the HuBERT framework. Experiment results on the SUPERB benchmark show that the proposed system achieves state-of-the-art performance in universal representation learning, especially for speaker identification oriented tasks. An ablation study is performed verifying the efficacy of each proposed method. Finally, we scale up the training dataset to 94 thousand hours of public audio data and achieve further performance improvement in all SUPERB tasks.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yoshua Bengio
2. Hung-yi Lee
3. Dong Yu
4. James R. Glass
5. Shinji Watanabe
6. Jinyu Li
7. Mohit Bansal
8. Tara N. Sainath

In Representation & Self-Supervised Learning

1. Hung-yi Lee
2. Abdel-rahman Mohamed
3. Shang-Wen Li
4. Karen Livescu
5. Lars Maaløe
6. Christian Igel
7. Jakob Drachmann Havtorn
8. Joakim Edin

In Speech & Audio

1. Tara N. Sainath
2. Abdel-rahman Mohamed
3. Shinji Watanabe
4. Hung-yi Lee
5. Jinyu Li
6. Shang-Wen Li
7. Karen Livescu
8. Lars Maaløe

In Signal Processing

1. Dong Yu
2. Shinji Watanabe
3. Hung-yi Lee
4. Jinyu Li
5. Tara N. Sainath
6. Abdel-rahman Mohamed
7. Karen Livescu
8. Shang-Wen Li

Citations in the corpus

• Robust Speech Recognition via Large-Scale Weak Supervision (2022, methodology): “Whisper performs well on Multilingual LibriSpeech, out-performing XLS-R (Babu et al., 2021), mSLAM (Bapna et al., 2022), and Maestro (Chen et al., 2022b) in a zero-shot setting.”
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, neutral): “To better learn speaker characteristics, [45] proposed UniSpeech-SAT, which extends the HuBERT framework with speaker-aware pre-training.”
• Self-Supervised Speech Representation Learning: A Review (2022, methodology): “At the same time, we observe a growing industry investment in pre-training with larger datasets, e.g., CPC-8k (8k hours) for Bidir-CPC [123], LL (60k hours) for CPC modified [122], wav2vec 2.0 [125], and HuBERT [130], Alexa internal datasets (10k hours) for wav2vec-c [126], Didi internal datasets (1”