Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation

By Mingi Ji, Seungjae Shin, Seunghyun Hwang, Gibeom Park, Il-Chul Moon. 2021

180 citations recorded by the source corpus.

Knowledge distillation is a method of transferring the knowledge from a pretrained complex teacher model to a student model, so a smaller network can replace a large teacher network at the deployment stage. To reduce the necessity of training a large teacher model, the recent literatures introduced a self-knowledge distillation, which trains a student network progressively to distill its own knowledge without a pretrained teacher network. While Self-knowledge distillation is largely divided into a data augmentation based approach and an auxiliary network based approach, the data augmentation approach looses its local information in the augmentation process, which hinders its applicability to diverse vision tasks, such as semantic segmentation. Moreover, these knowledge distillation approaches do not receive the refined feature maps, which are prevalent in the object detection and semantic segmentation community. This paper proposes a novel self-knowledge distillation method, Feature Refinement via Self-Knowledge Distillation (FRSKD), which utilizes an auxiliary self-teacher network to transfer a refined knowledge for the classifier network. Our proposed method, FRSKD, can utilize both soft label and feature-map distillations for the self-knowledge distillation. Therefore, FRSKD can be applied to classification, and semantic segmentation, which emphasize preserving the local information. We demonstrate the effectiveness of FRSKD by enumerating its performance improvements in diverse tasks and benchmark datasets. The implemented code is available at https://github.com/MingiJi/FRSKD.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohammad Rastegari
2. Sachin Mehta
3. Oncel Tuzel
4. Hadi Pouransari
5. Ali Farhadi
6. Mehrdad Farajtabar
7. Jian Yang
8. Qian Zhang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Mohammad Rastegari
3. Chunhua Shen
4. Ming-Hsuan Yang
5. Xiaogang Wang
6. Jian Yang
7. Oncel Tuzel
8. Dacheng Tao

In Representation & Self-Supervised Learning

1. Chuanguang Yang
2. Helong Zhou
3. Ali Farhadi
4. Sachin Mehta
5. Mehrdad Farajtabar
6. Qian Zhang
7. Hadi Pouransari
8. Linhang Cai

In Efficiency & Compression

1. Oncel Tuzel
2. Hadi Pouransari
3. Ali Farhadi
4. Mohammad Rastegari
5. Sachin Mehta
6. Mehrdad Farajtabar
7. Qian Zhang
8. Renjie Song

Citations in the corpus

• Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “Numerous variants of KD have been proposed, including feature distillation [28, 73], iterative distillation [43, 67], and self-distillation [65, 44, 18, 29].”
• Curriculum Temperature for Knowledge Distillation (2022, background): “Most existing works (Tung and Mori 2019; Chen et al. 2020; Ji et al. 2021) ignore the flexible role of the temperature and empirically set it to a fixed value (e.g., 4).”
• MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition (2022, methodology): “FRSKD [18] generates a self-teacher network for itself to provide refined feature maps for feature distillation.”