Injecting Text in Self-Supervised Speech Pretraining

By Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Gary Wang, Pedro J. Moreno Mengibar. 2021

39 citations recorded by the source corpus.

Self-supervised pretraining for Automated Speech Recognition (ASR) has shown varied degrees of success. In this paper, we propose to jointly learn representations during pretraining from two different modalities: speech and text. The proposed method, tts4pretrain complements the power of contrastive learning in self-supervision with linguistic/lexical representations derived from synthesized speech, effectively learning from untranscribed speech and unspoken text. Lexical learning in the speech encoder is enforced through an additional sequence loss term that is coupled with contrastive loss during pretraining. We demonstrate that this novel pretraining method yields Word Error Rate (WER) reductions of 10% relative on the well-benchmarked, Librispeech task over a state-of-the-art baseline pretrained with wav2vec2.0 only. The proposed method also serves as an effective strategy to compensate for the lack of transcribed speech, effectively matching the performance of 5000 hours of transcribed speech with just 100 hours of transcribed speech on the AMI meeting transcription task. Finally, we demonstrate WER reductions of up to 15% on an inhouse Voice Search task over traditional pretraining. Incorporating text into encoder pretraining is complimentary to rescoring with a larger or in-domain language model, resulting in additional 6% relative reduction in WER.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Yu Zhang
3. Tara N. Sainath
4. Dong Yu
5. Yonghui Wu
6. Chung-Cheng Chiu
7. Shinji Watanabe
8. Ye Jia

In Representation & Self-Supervised Learning

1. Jinyu Li
2. Furu Wei
3. Yu Zhang
4. Ye Jia
5. Tom Ko
6. Ankur Bapna
7. Chung-Cheng Chiu
8. Tara N. Sainath

In Speech & Audio

1. Jinyu Li
2. Yu Zhang
3. Tara N. Sainath
4. Yonghui Wu
5. Chung-Cheng Chiu
6. Ye Jia
7. Bo Li
8. Heiga Zen

Citations in the corpus

• MAESTRO: Matched Speech Text Representations through Modality Matching (2022, methodology): “One approach uses multi-task training which trains a single model with different objectives for each modality [5, 10, 11], while another uses modality conversion where text is directly converted to speech [12] via text-to-speech (TTS) approaches or viceversa [13] prior to self-supervised training.”
• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, methodology): “Besides, SpeechT5 is significantly superior to SpeechNet (Chen et al., 2021b) and pre-trained models from SUPERB (Yang et al., 2021) and achieves the state-of-the-art performance (i.e., 96.49”
• Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages (2023, methodology): “There is extensive literature on pre-training [6,12,22–33] and self-training [8,34–44] for ASR.”
• SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training (2021, methodology): “One approach to utilize data across modalities could involve synthetically transforming the modality of the data; one example being Chen et al. (2021) where the authors utilize text-to-speech (TTS) to transform text data into speech, and utilize it for monomodal speech pre-training.”
• Joint Unsupervised and Supervised Training for Multilingual ASR (2021, background): “It is often adopted to pretrain a sequence-to-sequence model and facilitate downstream tasks [1, 2, 3].”