Knowledge Distillation for Sequence Model
By Mingkun Huang, Yongbin You, Zhehuai Chen, Yanmin Qian, Kai Yu. 2018
70 citations recorded by the source corpus.
Knowledge distillation , or teacher-student training , has been effectively used to improve the performance of a relatively simpler deep learning model (the student) using a more complex model (the teacher). It is usually done by minimizing the Kullback-Leibler divergence (KLD) between the output distributions of the student and the teacher at each frame. However, the gain from frame-level knowledge distillation is limited for sequence models such as Connectionist Temporal Classification (CTC), due to the mismatch between the sequence-level criterion used in teacher model training and the frame-level criterion used in distillation. In this paper, sequence-level knowledge distillation is proposed to achieve better distillation performance. Instead of calculating a teacher posterior distribution given the feature vector of the current frame, sequence training criterion is employed to calculate the posterior distribution given the whole utterance and the teacher model. Experiments are conducted on both English Switchboard corpus and a large Chinese corpus. The proposed approach achieves significant and consistent improvements over the traditional frame-level knowledge distillation using both labeled and unlabeled data.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
In Signal Processing
Citations in the corpus
- Learning with Recoverable Forgetting (2022, neutral): “It has been successfully conducted in deep model compression [61], incremental learning [37], continual learning [25,57] and other tasks other than classification [5,53,22,50,36,60,51,55,23].”
- Knowledge Distillation: A Survey (2020, methodology): “e-level KD methods, sequence-level KD can perform better in some sequence models for speech recognition, such as connectionist temporalclassification(CTC)(Wong and Gales, 2016; Takashima et al., 2018; Huang et al., 2018). In general, sequence-level KD is carried out under the hypothesis of the label ”
- Amalgamating Filtered Knowledge: Learning Task-customized Student from Multi-task Teachers (2019, background): “For example, [Chen et al., 2017] handles object detection and learns a student model with improved accuracy; [Huang et al., 2018] specializes in speech applications by conducting knowledge distillation on sequence data.”
- Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition (2020, methodology): “Temporal Sequence Distillation (TSD) (Huang et al. 2018) also employs a light-weight model to distill a long Figure 2: The framework of FPCD.”
- Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More (2019, methodology): “Apart from classification, knowledge distillation has been utilized in other tasks [2, 13, 36].”
- Safe Distillation Box (2021, methodology): “Huang et al. (2018) focus on sequence-level knowledge distillation and has achieved encouraging results on speech applications.”
- Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning (2021, methodology): “In speech recognition (Huang et al., 2018) and natural language processing (Wang et al.”
- Structure-Level Knowledge Distillation For Multilingual Sequence Labeling (2020, background): “…the performance of small models with the guidance of big models, with applications in natural language processing (Kim and Rush, 2016; Kuncoro et al., 2016; Tan et al., 2019; Clark et al., 2019; Sun et al., 2019), computer vision (Ba and Caruana, 2014) and speech recognition (Huang et al., 2018).”