Self-Distillation: Towards Efficient and Compact Neural Networks

By Linfeng Zhang, Chenglong Bao, Kaisheng Ma. 2021

319 citations recorded by the source corpus.

Remarkable achievements have been obtained by deep neural networks in the last several years. However, the breakthrough in neural networks accuracy is always accompanied by explosive growth of computation and parameters, which leads to a severe limitation of model deployment. In this paper, we propose a novel knowledge distillation technique named self-distillation to address this problem. Self-distillation attaches several attention modules and shallow classifiers at different depths of neural networks and distills knowledge from the deepest classifier to the shallower classifiers. Different from the conventional knowledge distillation methods where the knowledge of the teacher model is transferred to another student model, self-distillation can be considered as knowledge transfer in the same model - from the deeper layers to the shallow layers. Moreover, the additional classifiers in self-distillation allow the neural network to work in a dynamic manner, which leads to a much higher acceleration. Experiments demonstrate that self-distillation has consistent and significant effectiveness on various neural networks and datasets. On average, 3.49 and 2.32 percent accuracy boost are observed on CIFAR100 and ImageNet. Besides, experiments show that self-distillation can be combined with other model compression methods, including knowledge distillation, pruning and lightweight model design.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chuanguang Yang
2. Helong Zhou
3. Qian Zhang
4. Linhang Cai
5. Zhulin An
6. Xiang Zhi
7. Jiwen Wu
8. Yongjun Xu

In Computer Vision and Pattern Recognition

1. Chuanguang Yang
2. Helong Zhou
3. Qian Zhang
4. Linhang Cai
5. Zhulin An
6. Xiang Zhi
7. Jiwen Wu
8. Yongjun Xu

Citations in the corpus

• MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition (2022, background): “Existing Self-KD works often employ auxiliary architecture [31,52,53,18,51] or data augmentation [37,48] to capture additional knowledge to enhance the network.”