Distilling Object Detectors with Task Adaptive Regularization

By Ruoyu Sun, Fuhui Tang, Xiaopeng Zhang, H. Xiong, Qi Tian. 2020

73 citations recorded by the source corpus.

Current state-of-the-art object detectors are at the expense of high computational costs and are hard to deploy to low-end devices. Knowledge distillation, which aims at training a smaller student network by transferring knowledge from a larger teacher model, is one of the promising solutions for model miniaturization. In this paper, we investigate each module of a typical detector in depth, and propose a general distillation framework that adaptively transfers knowledge from teacher to student according to the task specific priors. The intuition is that simply distilling all information from teacher to student is not advisable, instead we should only borrow priors from the teacher model where the student cannot perform well. Towards this goal, we propose a region proposal sharing mechanism to interflow region responses between the teacher and student models. Based on this, we adaptively transfer knowledge at three levels, \emph{i.e.}, feature backbone, classification head, and bounding box regression head, according to which model performs more reasonably. Furthermore, considering that it would introduce optimization dilemma when minimizing distillation loss and detection loss simultaneously, we propose a distillation decay strategy to help improve model generalization via gradually reducing the distillation penalty. Experiments on widely used detection benchmarks demonstrate the effectiveness of our method. In particular, using Faster R-CNN with FPN as an instantiation, we achieve an accuracy of $39.0\%$ with Resnet-50 on COCO dataset, which surpasses the baseline $36.3\%$ by $2.7\%$ points, and even better than the teacher model with $38.5\%$ mAP.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yunhe Wang
2. Kai Han
3. Jianyuan Guo
4. Chunjing Xu
5. Xinghao Chen
6. Han Wu
7. Yuan Gong
8. Yu Li

In Media Technology

1. Chunjing Xu
2. Yunhe Wang
3. Yu Li
4. Jianyuan Guo
5. Kai Han

Citations in the corpus

• Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation (2022, background): “For example, [5] and [32] distilled all components of the detector.”
• Focal and Global Knowledge Distillation for Detectors (2021, methodology): “[25] utilize the Gaussian Mask to cover the groundtruth for distillation.”
• Distilling Object Detectors via Decoupled Features (2021, methodology): “We report the results on mmdetection (ss-train, 12 epoch, and 1000×600 size) for fair comparisons with FGFI and TADF here.”
• Effective Whole-body Pose Estimation with Two-stages Distillation (2023, methodology): “Inspired by a detection distillation method TADF [41], we apply a weight-decay strategy for the distillation to reduce the distillation penalty gradually.”