Transferring knowledge from a RNN to a DNN

By William Chan, Nan Rosemary Ke, Ian Lane. 2015

77 citations recorded by the source corpus.

Deep Neural Network (DNN) acoustic models have yielded many state-of-the-art results in Automatic Speech Recognition (ASR) tasks. More recently, Recurrent Neural Network (RNN) models have been shown to outperform DNNs counterparts. However, state-of-the-art DNN and RNN models tend to be impractical to deploy on embedded systems with limited computational capacity. Traditionally, the approach for embedded platforms is to either train a small DNN directly, or to train a small DNN that learns the output distribution of a large DNN. In this paper, we utilize a state-of-the-art RNN to transfer knowledge to small DNN. We use the RNN model to generate soft alignments and minimize the Kullback-Leibler divergence against the small DNN. The small DNN trained on the soft RNN alignments achieved a 3.93 WER on the Wall Street Journal (WSJ) eval92 task compared to a baseline 4.54 WER or more than 13% relative improvement.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Sequence-Level Knowledge Distillation (2016, background): “There has also been work on transferring knowledge across different network architectures: Chan et al. (2015b) show that a deep non-recurrent neural network can learn from an RNN; Geras et al. (2016) train a CNN to mimic an LSTM for speech recognition.”
Knowledge Distillation: A Survey (2020, methodology): “…knowledge distillation systems for designing lightweight deep acoustic models for speech recognition (Chebotar and Waters 2016; Wong and Gales 2016; Chan et al. 2015; Price et al. 2016; Fukuda et al. 2017; Bai et al. 2019; Ng et al. 2018; Albanie et al. 2018; Lu et al. 2017; Shi et al. 2019a;…”
Policy Distillation (2015, methodology): “The contribution of this work is to describe and discuss the policy distillation approach and to demonstrate results on (a) single game distillation, (b) single game distillation with highly compressed models, (c) multi-game distillation, and (d) online distillation.”
Blending LSTMs into CNNs (2015, background): “The most similar is by Chan et al. (2015) who compressed LSTMs into small DNNs without convolutional layers.”
Improved training for online end-to-end speech recognition systems (2017, unknown): “This approach has been shown to be successful in several tasks including acoustic modeling [20], speech enchancement [21], and domain adaptation [22].”
Knowledge distillation across ensembles of multilingual models for low-resource languages (2017, methodology): “[5, 6] follows the KL-divergence criterion but used only a small portion (about 1%) of the top labels”
Learning Deep Representations with Probabilistic Knowledge Transfer (2018, methodology): “Also, in [6], the knowledge is transferred from a recurrent neural network (RNN) to a deep neural network using a similar approach.”
Recent progresses in deep learning based acoustic models (2017, background): “The second way is to employ teacher-student (T/S) learning or knowledge distillation, such as the works in [168]–[171].”
Generative Knowledge Transfer for Neural Language Models (2016, background): “Also, the model change from a fully connected deep neural network (FCDNN) to an RNN or the opposite direction has been tried (Tang et al., 2016; Chan et al., 2015).”
Conditional Teacher-student Learning (2019, background): “But in speech area, T is normally fixed at 1 [9, 13, 14].”
Do Deep Convolutional Nets Really Need to be Deep and Convolutional? (2016, methodology): “Readers familiar with distillation (model compression), training deep models on CIFAR-10, data augmentation, and Bayesian hyperparameter optimization may wish to skip to the empirical results in Section 3.”
Probabilistic Knowledge Transfer for Lightweight Deep Representation Learning (2020, methodology): “A large portion of the KT methods proposed in the literature employ the teacher model to generate soft-labels and then use these soft-labels for training the smaller student network [8], [9], [27]–[29].”
Personalized speech recognition on mobile devices (2016, background): “For embedded speech recognition, some authors have avoided RNNs citing increased computational costs and instead evaluated methods for transferring knowledge from RNNs to DNNs [11].”
Semi-supervised ensemble DNN acoustic model training (2017, background): “There is past work showing that it is possible to train a small DNN model to learn the predictions of a large DNN model [20] or RNN model [21] by minimizing the KL divergence between the output distributions on unlabeled data.”
Distilling Word Embeddings: An Encoding Approach (2015, background): “…in the literature shows the feasibility of transferring knowledge from one neural network to another, for instance, from shallow networks to deep ones (Romero et al., 2014), from feed-forward networks to recurrent ones (Wang et al., 2015), or vice versa (Ba and Caruana, 2014; Chan et al., 2015).”
Distilling Knowledge from Ensembles of Neural Networks for Speech Recognition (2016, background): “Similarly, Chan et al [11] distill knowledge from a RNN to a DNN, which likewise improves the WER of the student model.”