Regularizing Class-Wise Predictions via Self-Knowledge Distillation

By Sukmin Yun, Jongjin Park, Kimin Lee, Jinwoo Shin. 2020

350 citations recorded by the source corpus.

Deep neural networks with millions of parameters may suffer from poor generalization due to overfitting. To mitigate the issue, we propose a new regularization method that penalizes the predictive distribution between similar samples. In particular, we distill the predictive distribution between different samples of the same label during training. This results in regularizing the dark knowledge (i.e., the knowledge on wrong predictions) of a single network (i.e., a self-knowledge distillation) by forcing it to produce more meaningful and consistent predictions in a class-wise manner. Consequently, it mitigates overconfident predictions and reduces intra-class variations. Our experimental results on various image classification tasks demonstrate that the simple yet powerful method can significantly improve not only the generalization ability but also the calibration performance of modern convolutional neural networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianping Gou
2. Dacheng Tao
3. Lan Du
4. Liyuan Sun
5. Chuanguang Yang
6. Helong Zhou
7. Il-Chul Moon
8. Qian Zhang

In Computer Vision and Pattern Recognition

1. Jianping Gou
2. Qian Zhang
3. Dacheng Tao
4. Lan Du
5. Liyuan Sun
6. Chuanguang Yang
7. Helong Zhou
8. Il-Chul Moon

Citations in the corpus

• Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, background): “Self-distillation is a novel training scheme for knowledge transfer [23, 24, 27, 11, 10].”
• MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition (2022, methodology): “Although previous CS-KD and FRSKD also attempted to combine Mixup to further enhance the performance, they often utilized a straightforward instead of a thoughtful way.”
• Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation (2021, methodology): “Moreover, there are literatures on a self-knowledge distillation, which trains the student network progressively to distill and to regularize its own knowledge without the pretrained teacher network [43, 35, 18, 32, 39].”