Lessons from Building Acoustic Models with a Million Hours of Speech

By Sree Hari Krishnan Parthasarathi, Nikko Ström. 2019

90 citations recorded by the source corpus.

This is a report of our lessons learned building acoustic models from 1 Million hours of unlabeled speech, while labeled speech is restricted to 7,000 hours. We employ student/teacher training on unlabeled data, helping scale out target generation in comparison to confidence model based methods, which require a decoder and a confidence model. To optimize storage and to parallelize target generation, we store high valued logits from the teacher model. Introducing the notion of scheduled learning, we interleave learning on unlabeled and labeled data. To scale distributed training across a large number of GPUs, we use BMUF with 64 GPUs, while performing sequence training only on labeled data with gradient threshold compression SGD using 16 GPUs. Our experiments show that extremely large amounts of data are indeed useful; with little hyper-parameter tuning, we obtain relative WER improvements in the 10 to 20% range, with higher gains in noisier conditions.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “[50] used knowledge distillation on unlabeled data to teach a small student model for speech recognition.”
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, neutral): “In ASR, semi-supervised pseudo-label-style self-training has been explored generally in end-to-end settings in [45, 30, 25] for both low-resource [49, 11] and large-scale [40] setups.”
Self-Training and Pre-Training are Complementary for Speech Recognition (2020, positive): “This includes classical self-training [9, 10, 11] which demonstrated strong results [12, 13, 2, 14, 15] by pseudolabeling unannotated audio data and then retraining the final system with the additional labeled data.”
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, methodology): “Self-training has been a popular technique in ASR studied extensively in the literature [25–33].”
Improved Noisy Student Training for Automatic Speech Recognition (2020, background): “Self-training has a long history of being applied to ASR through numerous studies, e.g., [12, 13, 14, 15, 16, 17, 18, 19].”
Effectiveness of self-supervised pre-training for speech recognition (2019, background): “The practical need for building ASR systems for new conditions with limited resources spurred a lot of work focused on unsupervised speech recognition and representation learning [5, 6, 7, 8, 9, 10, 11, 12], in addition to semi- and weakly-supervised learning techniques to reduce the supervised data”
Self-Training for End-to-End Speech Recognition (2019, methodology): “Training with pseudo-labels can give an improvement to WER not only for low-resource languages [6, 7] but also on large-scale data sets [21].”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, methodology): “Section 4 describes the experimental setup, followed by our results and analysis in Section 5 where we apply pre-trained w2v-BERT models to LibriSpeech and voice search ASR.”
Rethinking Pre-training and Self-training (2020, background): “Recently, self-training has shown signiﬁcant progress in deep learning (e.g., image classiﬁcation [9,10], machine translation [11], and speech recognition [12,43]).”
Deep Contextualized Acoustic Representations for Semi-Supervised Speech Recognition (2019, methodology): “The most commonly-used SSL approach in ASR is self-training [1, 2, 3, 4, 5].”
Semi-Supervised Speech Recognition via Local Prior Matching (2020, background): “Knowledge distillation (KD) (Cui et al., 2017; Parthasarathi & Strom, 2019) and weak distillation (also known as selftraining or pseudo-labeling) (Veselỳ et al., 2017; Manohar et al., 2018; Li et al., 2019; Kahn et al., 2019a) have also achieved great success in semi-supervised learning for ASR.”
WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition (2021, background): “Recently, self-training [77, 79, 42, 43], a standard approach in semi-supervised learning [48, 83], is explored to signiﬁcantly boost the performance of image classiﬁcation.”
Training Keyword Spotting Models on Non-IID Data with Federated Learning (2020, background): “Given the obstacles to on-device labeling, teacher student training can be used to adapt a model trained on the server (with manually labeled data) to the on-device unlabeled data domain [42, 43, 44].”
Semi-supervised ASR by End-to-end Self-training (2020, background): “It has been shown to be very effective with traditional ASR pipelines [1, 2, 3, 4].”
slimIPL: Language-Model-Free Iterative Pseudo-Labeling (2020, background): “For competitive results in ASR, a language model (LM) was a key component of pseudo-labeling: it is usually combined with the acoustic model via beam-search decoding [4, 5, 6] or through shallow fusion [7, 8, 9, 10] to generate PLs.”
Unsupervised Prompt Learning for Vision-Language Models (2022, methodology): “Note that CLIP (Radford et al. 2021) utilizes the BPE representation (Sennrich, Haddow, and Birch 2015) for tokenization, so the vocabulary includes subwords that frequently appear in the text.”
Effectiveness of Self-Supervised Pre-Training for ASR (2020, background): “The practical need for building ASR systems for new conditions with limited resources spurred a lot of work focused on unsupervised speech recognition and representation learning [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], in addition to semi- and weakly-supervised learning techniques to reduce th”