Learning Efficient Object Detection Models with Knowledge Distillation

By Guobin Chen, Wongun Choi, Xiang Yu, Tony X. Han, Manmohan Chandraker. 2017

1,151 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Designing Interpretable Approximations to Deep Reinforcement Learning with Soft Decision Trees (2020, neutral): “For some applications, the size or complexity of such DNNs precludes their use in real time, or employment in edge devices with limited processing resources (Chen et al. 2017), (Mirzadeh et al.”
Learning with Recoverable Forgetting (2022, positive): “It has been successfully conducted in deep model compression [61], incremental learning [37], continual learning [25,57] and other tasks other than classification [5,53,22,50,36,60,51,55,23].”
Knowledge Distillation: A Survey (2020, background): “For example, the response in object detection task may contain the logits together with the offset of a bounding box (Chen et al., 2017).”
Faster ILOD: Incremental Learning for Object Detectors based on Faster RCNN (2020, methodology): “We use Faster RCNN (Ren et al., 2015) (Chen et al., 2017).”
Relational Knowledge Distillation (2019, methodology): “With growing interests in KD, task-specific KD methods have been proposed for object detection [5, 6, 37], face model compression [24], and image retrieval and Re-ID [7].”
Improved Knowledge Distillation via Teacher Assistant (2019, background): “Since then, knowledge distillation has been widely adopted in a variety of learning tasks (Yim et al. 2017; Yu et al. 2017; Schmitt et al. 2018; Chen et al. 2017).”
Learning Metrics From Teachers: Compact Networks for Image Embedding (2019, methodology): “Most literature on network distillation focuses on image classification, but recently several works have investigated applying the theory to object detection [3] and pedestrian detection [28].”
BlockDrop: Dynamic Inference Paths in Residual Networks (2017, methodology): “The need to deploy top-performing deep neural network models on mobile devices motivates techniques that can effectively reduce the storage and computational costs of such networks, including knowledge distillation [19, 40, 4], low-rank factorization [25, 47, 41], filter pruning [30, 36, 32, 57], qu”
Continual Universal Object Detection (2020, neutral): “Due to the simple implementation and success of knowledge distillation, the idea has been applied to different applications, such as object detection [4], and continual learning [6, 16, 26].”
KDGAN: Knowledge Distillation with Generative Adversarial Networks (2018, methodology): “[10] apply KD and hint learning to object detection problems.”
Distilling Object Detectors with Task Adaptive Regularization (2020, background): “[2] tries to distill the student model from all components including feature maps, classification head and bounding box regression head but the imitation of entire feature maps also introduces too many backgrounds which would drown the positive samples.”
Localization Distillation for Object Detection (2022, methodology): “Pseudo BBox Regression: The teacher bounded regression (TBR) loss [6] is a preliminary attempt to enhance the student on the localization head, i.”
Efficient Low-Resolution Face Recognition via Bridge Distillation (2020, background): “Note that it differs from many teacher-student frameworks [16]–[18], [20], [41], [42], where training is restricted on the private dataset or the same resolution or available target dataset.”
Relationship Matters: Relation Guided Knowledge Transfer for Incremental Learning of Object Detectors (2020, methodology): “With growing interests in knowledge distillation, task-speciﬁc KD methods have also been proposed for object detection [39, 4, 40, 29] but with ﬁxed number of classes.”
Channel-wise Knowledge Distillation for Dense Prediction* (2020, background): “The loss weight is set to 1, 2, 3, and T ∈ [1, 5].”
On Knowledge distillation from complex networks for response prediction (2019, background): “This idea of Knowledge Distil-lation has also been tried in the context of pruning networks for multiple object detection (Chen et al., 2017), speech recognition (Wong and Gales, 2016).”