Knowledge distillation for small-footprint highway networks

By Liang Lu, Michelle Guo, S. Renals. 2016

78 citations recorded by the source corpus.

Deep learning has significantly advanced state-of-the-art of speech recognition in the past few years. However, compared to conventional Gaussian mixture acoustic models, neural network models are usually much larger, and are therefore not very deployable in embedded devices. Previously, we investigated a compact highway deep neural network (HDNN) for acoustic modelling, which is a type of depth-gated feedforward neural network. We have shown that HDNN-based acoustic models can achieve comparable recognition accuracy with much smaller number of model parameters compared to plain deep neural network (DNN) acoustic models. In this paper, we push the boundary further by leveraging on the knowledge distillation technique that is also known as teacher-student training, i.e., we train the compact HDNN model with the supervision of a high accuracy cumbersome model. Furthermore, we also investigate sequence training and adaptation in the context of teacher-student training. Our experiments were performed on the AMI meeting speech recognition corpus. With this technique, we significantly improved the recognition accuracy of the HDNN acoustic model with less than 0.8 million parameters, and narrowed the gap between this model and the plain DNN with 30 million parameters.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Zhe Gan
3. Dong Yu
4. Yifan Gong
5. Jingjing Liu
6. Li Deng
7. Yu Cheng
8. Zhong Meng

In Signal Processing

1. Jinyu Li
2. Yifan Gong
3. Dong Yu
4. Zhong Meng
5. Yong Zhao
6. Yanmin Qian
7. Li Deng
8. M. Seltzer

Citations in the corpus

• Patient Knowledge Distillation for BERT Model Compression (2019, methodology): “Different from previous knowledge distillation methods (Hinton et al., 2015; Sau and Balasub-ramanian, 2016; Lu et al., 2017), we adopt a patient learning mechanism: instead of learning parameters from only the last layer of the teacher, we encourage the student model to extract knowledge also from…”
• Distilling Neural Networks for Greener and Faster Dependency Parsing (2020, methodology): “Teacherstudent distillation has successfully been exploited in NLP for machine translation, language modelling, and speech recognition (Kim and Rush, 4 2016; Yu et al., 2018; Lu et al., 2017).”
• An Investigation of a Knowledge Distillation Method for CTC Acoustic Models (2018, methodology): “The effectiveness of KD has been confirmed in speech recognition tasks [20, 21, 22, 23].”
• Knowledge Distillation for Sequence Model (2018, methodology): “In [9], the knowledge distillation can be combined with newly proposed parameter-efficient neural network structures.”
• Large-Scale Domain Adaptation via Teacher-Student Learning (2017, result): “This is consistent with the findings reported in [15][27].”
• Knowledge Distillation via Instance Relationship Graph (2019, background): “Therefore, more and more efforts have been putting into Knowledge distillation [9, 26, 16, 25] is one of the most popular solutions for model compression.”
• Conditional Teacher-student Learning (2019, result): “We obtain the best performance when T = 1 and the same conclusion is also reported in [15, 6].”