Self-Knowledge Distillation in Natural Language Processing

By Sangchul Hahn, Heeyoul Choi. 2019

135 citations recorded by the source corpus.

Since deep learning became a key player in natural language processing (NLP), many deep learning models have been showing remarkable performances in a variety of NLP tasks. Such high performance can be explained by efficient knowledge representation of deep learning models. Knowledge distillation from pretrained deep networks suggests that we can use more information from the soft target probability to train other neural networks. In this paper, we propose a self-knowledge distillation method, based on the soft target probabilities of the training model itself, where multimode information is distilled from the word embedding space right below the softmax layer. Due to the time complexity, our method approximates the soft target probabilities. In experiments, we applied the proposed method to two different and fundamental NLP tasks: language model and neural machine translation. The experiment results show that our proposed method improves performance on the tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Tao Qin
3. Tie-Yan Liu
4. Lijun Wu
5. Quoc V. Le
6. Xu Tan
7. Jianping Gou
8. Andrew G. Howard

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Xiaogang Wang
4. Quoc V. Le
5. Trevor Darrell
6. Tao Qin
7. Ming-Hsuan Yang
8. Yi Yang

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “The existing NLP tasks using KD contain neural machine translation (NMT) (Hahn and Choi, 2019; Zhou et al., 2019a; Li et al., 2021; Kim and Rush, 2016; Gordon and Duh, 2019; Tan et al., 2019; Wei et al., 2019; Freitag et al., 2017; Zhang et al., 2021b), text generation (Chen et al.”
• Large-Scale Generative Data-Free Distillation (2020, methodology): “The knowledge distillation approach has been an instrumental technique for model compression [6, 19], as well as for pushing state of the art by harnessing large amounts of unlabeled data to improve the performance of student models such as Noisy Student [54] and many others, see [52] for a survey.”
• R-Drop: Regularized Dropout for Neural Networks (2021, methodology): “Different from existing method that exploits dark knowledge from the model itself [20, 19] or distills knowledge between different layers [75], our strategy can be regarded as an instance-wise self-knowledge distillation, i.e., each pair of sampled sub models perform distillation between each other…”