Distilling Object Detectors With Fine-Grained Feature Imitation

By Tao Wang, Li Yuan, Xiaopeng Zhang, Jiashi Feng. 2019

476 citations recorded by the source corpus.

State-of-the-art CNN based recognition models are often computationally prohibitive to deploy on low-end devices. A promising high level approach tackling this limitation is knowledge distillation, which let small student model mimic cumbersome teacher model's output to get improved generalization. However, related methods mainly focus on simple task of classification while do not consider complex tasks like object detection. We show applying the vanilla knowledge distillation to detection model gets minor gain. To address the challenge of distilling knowledge in detection model, we propose a fine-grained feature imitation method exploiting the cross-location discrepancy of feature response. Our intuition is that detectors care more about local near object regions. Thus the discrepancy of feature response on the near object anchor locations reveals important information of how teacher model tends to generalize. We design a novel mechanism to estimate those locations and let student model imitate the teacher on them to get enhanced performance. We first validate the idea on a developed lightweight toy detector which carries simplest notion of current state-of-the-art anchor based detection models on challenging KITTI dataset, our method generates up to 15% boost of mAP for the student model compared to the non-imitated counterpart. We then extensively evaluate the method with Faster R-CNN model under various scenarios with common object detection benchmark of Pascal VOC and COCO, imitation alleviates up to 74% performance drop of student model compared to teacher. Codes released at https://github.com/twangnh/Distilling-Object-Detectors

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “…extended to other visual recognition applications, including face recognition (Luo et al. 2016; Kong et al. 2019; Yan et al. 2019; Ge et al. 2018; Wang et al. 2018b, 2019c; Duong et al. 2019; Wu et al. 2020; Wang et al. 2017; Zhang et al. 2020b; Wang et al. 2020b), image/video segmentation (He…”
Localization Distillation for Object Detection (2022, methodology): “Fine-Grained [66] focusing on the locations near GT boxes, produces 41.1 AP, which is comparable to the results of feature imitation using the Main Region.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Method GAN type Purpose Inputs of D Number of D Online KD Chen [31] First type Classification Logits One No Belagiannis [19] First type Classification Features One No Liu [135] First type ClassificationObject detection Features One No Hong [89] First type Object detection Features Six No Wang [229] ”
Channel-wise Knowledge Distillation for Dense Prediction* (2020, background): “[32] propose to distill the fine-grained feature near object anchor locations.”
Revisit Knowledge Distillation: a Teacher-free Framework (2019, unknown): “Knowledge Distillation Since Hinton et al. (2015) proposed knowledge distillation, KD has been widely adopted or modiﬁed (Romero et al., 2014; Yim et al., 2017; Yu et al., 2017; Furlanello et al., 2018; Anil et al., 2018; Mirzadeh et al., 2019; Wang et al., 2019).”
General Instance Distillation for Object Detection (2021, unknown): “Wang et al. [34] proposes a ﬁne-grained feature imitation for anchor-based detectors, distilling the near objects regions which are calculated by the intersection between GT boxes and anchors generated from detectors.”
Prime-Aware Adaptive Distillation (2020, methodology): “ROI mimic distillation [21] and fine-grained feature imitation [34] were designed for detection tasks by distilling the regions of interest.”
Distilling Object Detectors via Decoupled Features (2021, methodology): “We report the results on mmdetection (ss-train, 12 epoch, and 1000×600 size) for fair comparisons with FGFI and TADF here.”
Interactive Knowledge Distillation (2020, background): “To reduce the resource consumption while maintaining good performance, researchers leverage knowledge distillation techniques [6], [7], [8], [9], [10], [11] to transfer informative knowledge from a cumbersome but well-trained teacher network into a light-weight but unskilled student network.”
Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution (2020, methodology): “KD has been implemented in many machine learning tasks, for example image classiﬁcation [25], object detection [10, 61], video labeling [69], natural language processing [57, 38, 54, 34, 58], and speech recognition [9, 56, 35].”
Distilling Object Detectors with Task Adaptive Regularization (2020, methodology): “[28] propose to distill the backbone with features on local regions near object, and have demonstrated the power for detector distillation.”
A Simple Baseline for Pose Tracking in Videos of Crowed Scenes (2020, methodology): “Since RCNN [5] has been proposed, the two-stage detection methods have been widely adopted or modified [1, 9, 11, 17, 21, 22, 28].”
Relationship Matters: Relation Guided Knowledge Transfer for Incremental Learning of Object Detectors (2020, methodology): “With growing interests in knowledge distillation, task-speciﬁc KD methods have also been proposed for object detection [39, 4, 40, 29] but with ﬁxed number of classes.”
Continual Universal Object Detection (2020, unknown): “Top-down attention masks were ﬁrst proposed in [34] to do distillation on object detection, although it does not focus on avoiding catastrophic forgetting.”