Knowledge Distillation for Recurrent Neural Network Language Modeling with Trust Regularization
By Yangyang Shi, Mei-Yuh Hwang, X. Lei, Haoyu Sheng. 2019
27 citations recorded by the source corpus.
Recurrent Neural Networks (RNNs) have dominated language modeling because of their superior performance over traditional N-gram based models. In many applications, a large Recurrent Neural Network language model (RNNLM) or an ensemble of several RNNLMs is used. These models have large memory footprints and require heavy computation. In this paper, we examine the effect of applying knowledge distillation in reducing the model size for RNNLMs. In addition, we propose a trust regularization method to improve the knowledge distillation training for RNNLMs. Using knowledge distillation with trust regularization, we reduce the parameter size to a third of that of the previously published best model while maintaining the state-of-the-art perplexity result on Penn Treebank data. In a speech recognition N-best rescoring task, we reduce the RNNLM model size to 18.5% of the baseline system, with no degradation in word error rate (WER) performance on Wall Street Journal data set.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dacheng Tao
- Jianping Gou
- S. Maybank
- Baosheng Yu
- Subhabrata Mukherjee
- Ahmed Hassan Awadallah
- Shaohua Wan
- Qingqi Pei
Citations in the corpus
- Knowledge Distillation: A Survey (2020, methodology): “…et al. 2018, 2019c, 2020), audio classification (Gao et al. 2019; Perez et al. 2020), text-independent speaker recognition (Ng et al. 2018), speech enhancement (Watanabe et al. 2017), acoustic event detection (Price et al. 2016; Shi et al. 2019a,b), speech synthesis (Oord et al. 2018) and so on.”
- An Edge Traffic Flow Detection Scheme Based on Deep Learning in an Intelligent Transportation System (2021, methodology): “To do so, various structure compression and algorithm acceleration models were proposed, e.g., parameter pruning [28], parameter sharing [29], low-rank decomposition [30], compact convolution kernel [31] and knowledge distillation [32].”
- XtremeDistil: Multi-stage Distillation for Massive Multilingual Models (2020, background): “There has been some limited exploration of such techniques for sequence tagging tasks like NER (Izsak et al., 2019; Shi et al., 2019) or multilingual tasks (Tsai et al., 2019).”