Knowledge Distillation Meets Self-Supervision

By Guodong Xu, Ziwei Liu, Xiaoxiao Li, Chen Change Loy. 2020

341 citations recorded by the source corpus.

Knowledge distillation, which involves extracting the "dark knowledge" from a teacher network to guide the learning of a student network, has emerged as an important technique for model compression and transfer learning. Unlike previous works that exploit architecture-specific cues such as activation and attention for distillation, here we wish to explore a more general and model-agnostic approach for extracting "richer dark knowledge" from the pre-trained teacher model. We show that the seemingly different self-supervision task can serve as a simple yet powerful solution. For example, when performing contrastive learning between transformed entities, the noisy predictions of the teacher network reflect its intrinsic composition of semantic and pose information. By exploiting the similarity between those self-supervision signals as an auxiliary task, one can effectively transfer the hidden information from the teacher to the student. In this paper, we discuss practical ways to exploit those noisy self-supervision signals with selective transfer for distillation. We further show that self-supervision signals improve conventional distillation with substantial gains under few-shot and noisy-label scenarios. Given the richer knowledge mined from self-supervision, our knowledge distillation approach achieves state-of-the-art performance on standard benchmarks, i.e., CIFAR100 and ImageNet, under both similar-architecture and cross-architecture settings. The advantage is even more pronounced under the cross-architecture setting, where our method outperforms the state of the art CRD by an average of 2.3% in accuracy rate on CIFAR100 across six different teacher-student pairs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Han-Jia Ye
3. De-chuan Zhan
4. Wei-Lun Chao
5. Lu Ming
6. Jing Zhang
7. Jianping Gou
8. Jinguo Zhu

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Han-Jia Ye
3. De-chuan Zhan
4. Wei-Lun Chao
5. Jing Zhang
6. Lu Ming
7. Jianping Gou
8. Jinguo Zhu

In Radiology, Nuclear Medicine and Imaging

1. Han-Jia Ye
2. De-chuan Zhan
3. Dacheng Tao
4. Jing Zhang
5. Dapeng Chen
6. S. Maybank
7. Troy Luhman
8. Eric Luhman

Citations in the corpus

• Knowledge Distillation: A Survey (2020, background): “…such as the adversarial learning (Liu et al. 2018), auto machine learning (Macko et al. 2019; Fakoor et al. 2020), label noise filtering learning (Xia et al. 2018), lifelong learning (Zhai et al. 2019), and reinforcement learning (Ashok et al. 2018; Xu et al. 2020c; Zhao and Hospedales 2020).”
• Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed (2021, background): “Knowledge distillation (Bucilua et al., 2006; Hinton et al., 2015) has been explored extensively in supervised learning tasks such as image classification [26, 46, 44, 7, 36, 43], language modeling [35, 37, 28, 14, 34], and speech recognition [3, 6, 40, 17, 9].”
• Few-Shot Learning With a Strong Teacher (2021, methodology): “We also note that, our method is different from self-distillation [36], [37], [38], [39] — our teacher is a classifier trained with ample data while our student is a few-shot learner.”
• Cross-Layer Distillation with Semantic Calibration (2020, methodology): “In contrast to pairwise transfer, some approaches formulate knowledge distillation from the perspective of contrastive learning to capture higher-order dependencies in the representation space [44], [45].”
• Continual Contrastive Self-supervised Learning for Image Classification (2021, background): “In SSKD[56], the similarity matrix of feature representations of images in a mini-batch output by teacher network is imitated by the student.”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “, rotation, cropping, color permutation) [164, 166, 235], which has been mentioned in Sec.”
• A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “…monocular vision [183], person reidentification (re-ID) [184], [185], visual odometry [186], scene flow estimation [187], knowledge distillation [188], optical flow prediction [189], vision-language navigation [190], physiological signal estimation [191], [192], image denoising [193], [194],…”
• Complementary Relation Contrastive Distillation (2021, methodology): “5, we can observe that our distillation method CRCD can consistently outperform all other distillation methods with a large margin, including the recent state-of-the-arts, CRD and SSKD. Additionally, even only one element (feature or its gradient) is used in the relation distillation, our method can”