Prime-Aware Adaptive Distillation
By Youcai Zhang, Zhonghao Lan, Yuchen Dai, Fangao Zeng, Yan Bai, Jie Chang, Yichen Wei. 2020
53 citations recorded by the source corpus.
Knowledge distillation(KD) aims to improve the performance of a student network by mimicing the knowledge from a powerful teacher network. Existing methods focus on studying what knowledge should be transferred and treat all samples equally during training. This paper introduces the adaptive sample weighting to KD. We discover that previous effective hard mining methods are not appropriate for distillation. Furthermore, we propose Prime-Aware Adaptive Distillation (PAD) by the incorporation of uncertainty learning. PAD perceives the prime samples in distillation and then emphasizes their effect adaptively. PAD is fundamentally different from and would refine existing methods with the innovative view of unequal training. For this reason, PAD is versatile and has been applied in various tasks including classification, metric learning, and object detection. With ten teacher-student combinations on six datasets, PAD promotes the performance of existing distillation methods and outperforms recent state-of-the-art methods.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- Knowledge Distillation: A Survey (2020, methodology): “…mutual teacher–student networks (Thoker and Gall 2019), multiple stream networks (Garcia et al. 2018), spatiotemporal distilled dense-connectivity network (Hao and Zhang 2019), graph distillation (Luo et al. 2018) and multi-teacher to multi-student networks (Wu et al. 2019b; Zhang et al. 2020c).”
- Distilling Object Detectors via Decoupled Features (2021, methodology): “To be specific, PAD [68] uses a stronger baseline implemented on detectron2 (ms-train, 17 epoch, and 1200×800 size).”