Distilling Knowledge from Ensembles of Neural Networks for Speech Recognition

By Yevgen Chebotar, Austin Waters. 2016

174 citations recorded by the source corpus.

Speech recognition systems that combine multiple types of acoustic models have been shown to outperform single-model systems. However, such systems can be complex to implement and too resource-intensive to use in production. This paper describes how to use knowledge distillation to combine acoustic models in a way that has the best of many worlds: It improves recognition accuracy signiﬁcantly, can be implemented with standard training tools, and requires no additional complexity during recognition. First, we identify a simple but particularly strong type of ensemble: a late combination of recurrent neural networks with different architectures and training objectives. To harness such an ensemble, we use a variant of standard cross-entropy training to distill it into a single model and then discriminatively ﬁne-tune the result. An evaluation on 2,000-hour large vocabulary tasks in 5 languages shows that the distilled models provide up to 8.9% relative WER improvement over conventionally-trained baselines with an identical number of parameters.

Subfields: Artificial Intelligence, Efficiency & Compression, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

In Speech & Audio

In Signal Processing

Citations in the corpus

Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, background): “Ensemble models on the other hand, have been shown to be promising teachers from the early work of [9] until recent works in various domains [10, 68, 54, 56] and with techniques to boost the their performance [52, 17, 41].”
Knowledge Distillation: A Survey (2020, methodology): “There are many knowledge distillation systems for designing lightweight deep acoustic models for speech recognition (Chebotar and Waters 2016; Wong and Gales 2016; Chan et al. 2015; Price et al. 2016; Fukuda et al. 2017; Bai et al. 2019; Ng et al. 2018; Albanie et al. 2018; Lu et al. 2017; Shi et…”
Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed (2021, background): “Knowledge distillation (Bucilua et al., 2006; Hinton et al., 2015) has been explored extensively in supervised learning tasks such as image classiﬁcation [26, 46, 44, 7, 36, 43], language modeling [35, 37, 28, 14, 34], and speech recognition [3, 6, 40, 17, 9].”
On the Efficacy of Knowledge Distillation (2019, background): “In terms of applications of knowledge distillation, prior work has found knowledge distillation to be useful for sequence modeling [16, 4], semi-supervised learning [21], domain adaptation [19], multi-modal learning [9] and so on.”
Knowledge distillation across ensembles of multilingual models for low-resource languages (2017, background): “Knowledge distillation is an active area of research [5, 6, 7, 8, 9, 10].”
Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning (2020, background): “Nor can they explain why the superior performance of ensemble can also be distilled into a single model using knowledge distillation [21, 23, 30, 35, 43, 61, 91].”
Efficient Knowledge Distillation from an Ensemble of Teachers (2017, methodology): “In other approaches, [13] uses oracle to select the best teacher ensemble for each utterance while in [14] both temparature and multitasking are used to combine teacher labels and original hard labels.”
Empirical Analysis of Knowledge Distillation Technique for Optimization of Quantized Deep Neural Networks (2019, methodology): “The promising performance improvement of the knowledge distillation technique, it utilizes in variety of applications (Chebotar and Waters 2016; Dai and Van Gool 2018; Chen et al. 2017; Oord et al. 2017) and learning algorithms (Romero et al. 2014; Kulkarni, Patil, and Karande 2017; Park et al.…”
An Investigation of a Knowledge Distillation Method for CTC Acoustic Models (2018, methodology): “The effectiveness of KD has been confirmed in speech recognition tasks [20, 21, 22, 23].”
Stochasticity and Skip Connection Improve Knowledge Transfer (2021, background): “Multiple teacher networks provide comprehensive guidance to help the learning of student networks [8], [12].”
Probabilistic Knowledge Transfer for Lightweight Deep Representation Learning (2020, methodology): “Several KT methods have been proposed and successfully used for a wide variety of tasks, mainly related to classi-ﬁcation [7], [8], [10] and object detection [11].”
Reinforced Multi-Teacher Selection for Knowledge Distillation (2020, methodology): “W-Ensemble Teacher is a weighted assemble method following Chebotar and Waters (2016); Fukuda et al.”
Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution (2020, methodology): “KD has been implemented in many machine learning tasks, for example image classification [28], object detection [12, 61], video labeling [70], natural language processing [56, 41, 53, 36, 57], and speech recognition [11, 55, 37].”
Does Knowledge Distillation Really Work? (2021, background): “Other early work proposed distilling ensembles of shallow networks into a single network [42], an idea which resonates with more recent work on the distillation of deep ensembles [2, 7, 35, 38, 40].”
Compression of Acoustic Event Detection Models With Quantized Distillation (2019, methodology): “For instance, [10] evaluates the combination of low-rank factorization, knowledge distillation and network pruning in ASR.”