Residual Knowledge Distillation

By Mengya Gao, Yujun Shen, Quanquan Li, Chen Change Loy. 2020

32 citations recorded by the source corpus.

Knowledge distillation (KD) is one of the most potent ways for model compression. The key idea is to transfer the knowledge from a deep teacher model (T) to a shallower student (S). However, existing methods suffer from performance degradation due to the substantial gap between the learning capacities of S and T. To remedy this problem, this work proposes Residual Knowledge Distillation (RKD), which further distills the knowledge by introducing an assistant (A). Specifically, S is trained to mimic the feature maps of T, and A aids this process by learning the residual error between them. In this way, S and A complement with each other to get better knowledge from T. Furthermore, we devise an effective method to derive S and A from a given model without increasing the total computational cost. Extensive experiments show that our approach achieves appealing results on popular classification datasets, CIFAR-100 and ImageNet, surpassing state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Jianping Gou
3. S. Maybank
4. Lin Wang
5. Kuk-Jin Yoon
6. Baosheng Yu
7. Luc Van Gool
8. Trevor Darrell

In Radiology, Nuclear Medicine and Imaging

1. Dacheng Tao
2. S. Maybank
3. Lin Wang
4. Baosheng Yu
5. Jianping Gou
6. Kuk-Jin Yoon

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “, the assistant structure was used to learn the residual error (Gao et al., 2020).”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “Although [69] proposes to distill knowledge by directly matching feature maps, a teaching assistant is introduced to learn the residual errors of between the feature maps of the student and teacher.”