Adaptive Knowledge Distillation Based on Entropy

By Kisoo Kwon, Hwidong Na, Hoshik Lee, Nam Soo Kim. 2020

74 citations recorded by the source corpus.

Knowledge distillation (KD) approach is widely used in the deep learning field mainly for model size reduction. KD utilizes soft labels of teacher model, which contain the dark- knowledge that one-hot ground-truth does not have. This knowledge can improve the performance of already saturated student model. In case of multiple-teacher models, generally, the same weighted average (interpolated training) of multiple-teacher’s labels is applied to KD training. However, if the knowledge characteristics among teachers are somewhat different, the interpolated training can be at risk of crushing each knowledge characteristics and can also raise noise component. In this paper, we propose an entropy based KD training, which utilizes the teacher model labels with lower entropy at a larger rate among the various teacher models. The proposed method shows a better performance than the conventional KD training scheme in automatic speech recognition.

Subfields: Artificial Intelligence, Signal Processing

Citations in the corpus

• Knowledge Distillation: A Survey (2020, neutral): “…2019a; Roheda et al. 2018; Shi et al. 2019b; Gao et al. 2019; Ghorbani et al. 2018; Takashima et al. 2018; Watan-abe et al. 2017; Shi et al. 2019c; Asami et al. 2017; Huang et al. 2018; Shen et al. 2018; Perez et al. 2020; Shen et al. 2019c; Oord et al. 2018; Kwon et al. 2020; Shen et al. 2020).”