Decoupled Knowledge Distillation

By Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, Jiajun Liang. Computer Vision and Pattern Recognition, 2022

891 citations recorded by the source corpus.

State-of-the-art distillation methods are mainly based on distilling deep features from intermediate layers, while the significance of logit distillation is greatly overlooked. To provide a novel viewpoint to study logit distillation, we re-formulate the classical KD loss into two parts, i.e., target class knowledge distillation (TCKD) and non-target class knowledge distillation (NCKD). We empirically investigate and prove the effects of the two parts: TCKD transfers knowledge concerning the “difficulty” of training samples, while NCKD is the prominent reason why logit distillation works. More importantly, we reveal that the classical KD loss is a coupled formulation, which (1) suppresses the effectiveness of NCKD and (2) limits the flexibility to balance these two parts. To address these issues, we present Decoupled Knowledge Distillation (DKD), enabling TCKD and NCKD to play their roles more efficiently and flexibly. Compared with complex feature-based methods, our DKD achieves comparable or even better results and has better training efficiency on CIFAR-100, ImageNet, and MS-COCO datasets for image classification and object detection tasks. This paper proves the great potential of logit distillation, and we hope it will be helpful for future research. The code is available at https://github.com/megviiresearch/mdistiller.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/72b989a52a5cc2eee44bba29e8d225ce7bc07666

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yunchao Wei
2. Yao Zhao
3. Xiaodan Liang
4. Chunjie Zhang
5. Lianghua He
6. Renjie Song
7. Borui Zhao
8. Yu Li

In Computer Vision and Pattern Recognition

1. Yunchao Wei
2. Xiaodan Liang
3. Luc Van Gool
4. Yao Zhao
5. Jian Yang
6. Yi Yang
7. Ming-Hsuan Yang
8. Chunhua Shen

In Radiology, Nuclear Medicine and Imaging

1. Yu Li
2. Yao Zhao
3. Xiaodan Liang
4. Yunchao Wei
5. Jian Yang

Citations in the corpus

• Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective (2022, neutral): “The general framework could be similar to the well-known knowledge distillation process [18, 13, 53]; however, learning the first stage without the LIAS loss function (which corresponds to the teacher model training) still leads to poor transferability in T1.”
• EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything (2023, neutral): “Decoupled knowledge distillation[70] separates”
• SCConv: Spatial and Channel Reconstruction Convolution for Feature Redundancy (2023, background): “Knowledge distillation [11, 34] generates small student networks with the guidance of a well-trained big teacher network.”
• Curriculum Temperature for Knowledge Distillation (2022, background): “• DKD (Zhao et al. 2022): γ = 1 .”
• CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation (2023, background): “However, such a large similarity would suppress the relational distillation of other unmatched pairs [79].”
• Effective Whole-body Pose Estimation with Two-stages Distillation (2023, background): “Some following works [15, 52, 54] utilize teacher’s logits in different ways, transferring more knowledge from soft labels, target and non-target logits [16, 58, 64, 66, 58].”