A Comprehensive Overhaul of Feature Distillation

By Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, J. Choi. 2019

759 citations recorded by the source corpus.

We investigate the design aspects of feature distillation methods achieving network compression and propose a novel feature distillation method in which the distillation loss is designed to make a synergy among various aspects: teacher transform, student transform, distillation feature position and distance function. Our proposed distillation loss includes a feature transform with a newly designed margin ReLU, a new distillation feature position, and a partial L2 distance function to skip redundant information giving adverse effects to the compression of student. In ImageNet, our proposed method achieves 21.65% of top-1 error with ResNet50, which outperforms the performance of the teacher network, ResNet152. Our proposed method is evaluated on various tasks such as image classification, object detection and semantic segmentation and achieves a significant performance improvement in all tasks. The code is available at project page.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Transfer without Forgetting (2022, methodology): “where c is the current task and ReLUm(·) indicates the application of a margin ReLU activation [26].”
Knowledge Distillation: A Survey (2020, methodology): “…including the design of knowledge (Hinton et al. 2015; Romero et al. 2015) and the loss functions for matching features or distributions matching (Huang and Wang 2017; Passalis and Tefas 2018; Zagoruyko and Komodakis 2017; Mirzadeh et al. 2020; Li et al. 2020d; Heo et al. 2019b; Asif et al. 2020).”
Knowledge distillation via adaptive instance normalization (2020, methodology): “For the above mentioned settings, we compare our method with KD [5] and AT [6], and the more recent methods of OFD [22] from ICCV 2019, RKD [25] from CVPR2019, CRD [29] from ICLR2020.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “or signiﬁcant KD to the student. Another reason comes from when the student is unable to follow the teacher due to the large model capacity between the teacher and the student. It is stated in [1], [52] that the S-T similarity is highly related to how well the student can mimic the teacher. If the ”
Unifying Voxel-based Representation with Transformer for 3D Object Detection (2022, background): “3, the feature distance for knowledge transfer is formulated as dKT = PL2(TP (x, y, z),SI(x, y, z)), (3) where PL2 represents the partial L2 distance [40].”
Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels (2021, background): “Studies on KD have enriched and diversified the options for the teacher, such as feature map distillation [58, 19, 18], relation-based distillation [34, 47], ensemble distillation [40, 61], or iterative self-distillation [9, 55, 53].”
Differentiable Feature Aggregation Search for Knowledge Distillation (2020, methodology): “The general design schemes for feature distillation are categorized into teacher transform, student transform, distillation position and distance function [10].”
Self-Distillation as Instance-Specific Label Smoothing (2020, background): “For instance, distilling knowledge from intermediate hidden layers were found to be beneficial [14, 17, 18, 32, 34, 39].”
QKD: Quantization-aware Knowledge Distillation (2019, methodology): “The works [49, 20, 14, 37] have shown promising performance in the full-precision (FP) domain (teacher and student are both FP) by using of-ﬂine activation (feature map) distillation.”
Cross-Layer Distillation with Semantic Calibration (2020, methodology): “To preserve as much information as possible in the teacher model during knowledge transfer, previous works attempt to adjust the distillation position to the front of the ReLU operation [21], [64], which is called pre-activation distillation.”
Distilling Knowledge via Knowledge Review (2021, methodology): “Since only a few methods [31, 8] are claimed workable for detection, we reproduce the popular ones [9, 25] and the latest one [31].”
Focal and Global Knowledge Distillation for Detectors (2021, methodology): “There have been many works [10, 27, 33, 34] that improve the student classifiers significantly.”
Knowledge Distillation from A Stronger Teacher (2022, background): “For example, OFD [14] proposes to distill the information via multiple intermediate layers, but requires additional convolutions for feature alignments; CRD [41] introduces a contrastive loss to transfer pair-wise relationships, but it needs to hold a memory bank for all 128-d features of ImageNet i”
Matching Guided Distillation (2020, methodology): “Previous works [28,39,11] overcome these two obstacles by building an adaptive module between hidden layers of teacher and student.”
Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation (2021, methodology): “In this paper, we used an experimental setup similar to that in [Heo et al. , 2019a; Cho and Hariharan, 2019]: image classiﬁ-cation on CIFAR-100 with a family of Wide-ResNet (WRN) [ Zagoruyko and Komodakis, 2016b ] and ImageNet with a family of of ResNet (RN) [ He et al. , 2016 ] .”