Semi-supervised Training for End-to-end Models via Weak Distillation

By Bo Li, Tara N. Sainath, Ruoming Pang, Zelin Wu. 2019

54 citations recorded by the source corpus.

End-to-end (E2E) models are a promising research direction in speech recognition, as the single all-neural E2E system offers a much simpler and more compact solution compared to a conventional model, which has a separate acoustic (AM), pronunciation (PM) and language model (LM). However, it has been noted that E2E models perform poorly on tail words and proper nouns, likely because the end-to-end optimization requires joint audio-text pairs, and does not take advantage of additional lexicons and large amounts of text-only data used to train the LMs in conventional models. There has been numerous efforts in training an RNN-LM on text-only data and fusing it into the end-to-end model. In this work, we contrast this approach to training the E2E model with audio-text pairs generated from unsupervised speech data. To target the proper noun issue specifically, we adopt a Part-of-Speech (POS) tagger to filter the unsupervised data to use only those with proper nouns. We show that training with filtered unsupervised-data provides up to a 13% relative reduction in word-error-rate (WER), and when used in conjunction with a cold-fusion RNN-LM, up to a 17% relative improvement.

Subfields: Artificial Intelligence, Efficiency & Compression, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

In Speech & Audio

Citations in the corpus

Multitask Training with Text Data for End-to-End Speech Recognition (2020, methodology): “proposed to train the ASR model using audio-text pairs generated from text-only data [21].”
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, methodology): “Self-training has been a popular technique in ASR studied extensively in the literature [25–33].”
Improved Noisy Student Training for Automatic Speech Recognition (2020, background): “Self-training has a long history of being applied to ASR through numerous studies, e.g., [12, 13, 14, 15, 16, 17, 18, 19].”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, methodology): “Self-training has been a practically useful and extensively studied technique in ASR [4, 5, 6, 7, 8, 9].”
Streaming End-to-end Speech Recognition for Mobile Devices (2018, methodology): “Finally, in order to improve RNN-T performance on proper nouns, which is critical for biasing, we train with an additional 500M unsupervised voice search utterances (each training batch is filled with supervised data 80% of the time and unsupervised data 20% of the time) [39].”
Semi-Supervised Speech Recognition via Local Prior Matching (2020, background): “, 2017; Parthasarathi & Strom, 2019) and weak distillation (also known as selftraining or pseudo-labeling) (Veselỳ et al., 2017; Manohar et al., 2018; Li et al., 2019; Kahn et al., 2019a) have also achieved great success in semi-supervised learning for ASR.”
Effectiveness of Self-Supervised Pre-Training for ASR (2020, background): “The practical need for building ASR systems for new conditions with limited resources spurred a lot of work focused on unsupervised speech recognition and representation learning [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], in addition to semi- and weakly-supervised learning techniques to reduce th”
An Attention-Based Joint Acoustic and Text on-Device End-To-End Model (2020, methodology): “In addition, it does not require additional training stages, unlike past work with unsupervised data [14].”
Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “TTS audio sometimes also degrades the recognition accuracy on real speech [147].”
Contrastive Semi-Supervised Learning for ASR (2021, methodology): “Since the 1990s, pseudo-labeling showed improvements to classical HMM/GMM systems as well as hybrid-NN systems [2, 3, 4, 6, 5, 7], and also for end-to-end ASR approaches [8, 9, 10, 11, 12].”
Large scale weakly and semi-supervised learning for low-resource video ASR (2020, methodology): “On the other hand, seq2seq models showed greater grace working with noisy labels both for personal assistant domains [16] and audio books [49, 50].”