Focal and Global Knowledge Distillation for Detectors

By Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, Chun Yuan. Computer Vision and Pattern Recognition, 2021

385 citations recorded by the source corpus.

Knowledge distillation has been applied to image classification successfully. However, object detection is much more sophisticated and most knowledge distillation methods have failed on it. In this paper, we point out that in object detection, the features of the teacher and student vary greatly in different areas, especially in the foreground and background. If we distill them equally, the uneven differences between feature maps will negatively affect the distillation. Thus, we propose Focal and Global Distillation (FGD). Focal distillation separates the foreground and background, forcing the student to focus on the teacher's critical pixels and channels. Global distillation rebuilds the relation between different pixels and transfers it from teachers to students, compensating for missing global information in focal distillation. As our method only needs to calculate the loss on the feature map, FGD can be applied to various detectors. We experiment on various detectors with different backbones and the results show that the student detector achieves excellent mAP improvement. For example, ResNet-50 based RetinaNet, Faster RCNN, RepPoints and Mask RCNN with our distillation method achieve 40.7%, 42.0%, 42.0% and 42.1% mAP on COCO2017, which are 3.3, 3.6, 3.4 and 2.9 higher than the baseline, respectively. Our codes are available at https://github.com/yzd-v/FGD.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c5298a80a89a5a65489b42991f9a87f734d9e0b2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Chuanguang Yang
2. Helong Zhou
3. Qian Zhang
4. Linhang Cai
5. Yu Li
6. Chun Yuan
7. Zhulin An
8. Zhendong Yang

Citations in the corpus

• MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition (2022, background): “Online KD [54,57,3,41,40,45,39] aims to train multiple student networks and perform knowledge transfer within the cohort in an online manner.”
• Effective Whole-body Pose Estimation with Two-stages Distillation (2023, methodology): “From the logit-based distillation to feature-based distillation, the knowledge is transferred from intermediate layers [17, 55, 57] and it extends the distillation to various tasks, including detection [4, 56], segmentation [38], generation [30] and so on.”