UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data

By Chengyi Wang, Yuehua Wu, Yu Wu, Yao Qian, Ken'ichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, Xuedong Huang. 2021

143 citations recorded by the source corpus.

In this paper, we propose a unified pre-training approach called UniSpeech to learn speech representations with both unlabeled and labeled data, in which supervised phonetic CTC learning and phonetically-aware contrastive self-supervised learning are conducted in a multi-task learning manner. The resultant representations can capture information more correlated with phonetic structures and improve the generalization across languages and domains. We evaluate the effectiveness of UniSpeech for cross-lingual representation learning on public CommonVoice corpus. The results show that UniSpeech outperforms self-supervised pretraining and supervised transfer learning for speech recognition by a maximum of 13.4% and 17.8% relative phone error rate reductions respectively (averaged over all testing languages). The transferability of UniSpeech is also demonstrated on a domain-shift speech recognition task, i.e., a relative word error rate reduction of 6% against the previous approach.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, background): “on phoneme classification [4] and automatic speech recognition (ASR) [5]–[7].”
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “…and shown promising results, beneﬁting from richly learned representations (Chung and Glass, 2018; Chuang et al., 2020; Song et al., 2019; Baevski et al., 2020; Wang et al., 2021; Hsu et al., 2021; Chung et al., 2021a), such as wav2vec 2.0 (Baevski et al., 2020) and HuBERT (Hsu et al., 2021).”
AudioGen: Textually Guided Audio Generation (2022, methodology): “…Learning (SSL) methods for speech have shown remarkable results for automatic speech recognition (Schneider et al., 2019; Baevski et al., 2020; Wang et al., 2021), phoneme segmentation (Kreuk et al., 2020), and audio compression (Zeghidour et al., 2021; Polyak et al., 2021). van den Oord et…”
Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition (2021, background): “There has been a great deal of attention to multi-lingual automatic speech recognition (ASR) that can recognize multiple languages [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].”
Joint Unsupervised and Supervised Training for Multilingual ASR (2021, methodology): “To further increase the quantizer codebook usage, UniSpeech randomly replaces contextual representations with quantized latent codes.”
textless-lib: a Library for Textless Spoken Language Processing (2022, methodology): “This area of research has been made possible by converging progress in self-supervised speech representation learning (Schneider et al., 2019; Baevski et al., 2020; Oord et al., 2018; Hsu et al., 2021; Chorowski et al., 2021; Chen et al., 2021; Chung et al., 2021; Wang et al., 2021; Ao et al., 2021)”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, background): “In the literature, these are often jointly referred to as reconstruction tasks (REC, table 1) (Liu, Li, and Lee 2021; Wang et al. 2021), although this is somewhat misleading in the case of prediction.”