Distilling Knowledge via Knowledge Review

By Pengguang Chen, Shu Liu, Hengshuang Zhao, Jiaya Jia. 2021

661 citations recorded by the source corpus.

Knowledge distillation transfers knowledge from the teacher network to the student one, with the goal of greatly improving the performance of the student network. Previous methods mostly focus on proposing feature transformation and loss functions between the same level's features to improve the effectiveness. We differently study the factor of connection path cross levels between teacher and student networks, and reveal its great importance. For the first time in knowledge distillation, cross-stage connection paths are proposed. Our new review mechanism is effective and structurally simple. Our finally designed nested and compact framework requires negligible computation overhead, and outperforms other methods on a variety of tasks. We apply our method to classification, object detection, and instance segmentation tasks. All of them witness significant student network performance improvement.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chen Qian
2. Fei Wang
3. Shan You
4. Jiajun Liang
5. Quan Cui
6. Renjie Song
7. Tao Huang
8. Borui Zhao

In Computer Vision and Pattern Recognition

1. Chen Qian
2. Fei Wang
3. Shan You
4. Jiajun Liang
5. Quan Cui
6. Renjie Song
7. Tao Huang
8. Borui Zhao

Citations in the corpus

• Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation (2022, methodology): “Nevertheless, the nature of these methods are designed using feature-based knowledge distillation [6], response-based method [12] has not been explored in IOD [25] yet.”
• Knowledge Distillation from A Stronger Teacher (2022, background): “For example, CRD [41] needs to preserve a memory bank for all 128-d features of ImageNet images, and produces additional 260M FLOPs of computation cost; SRRL [47] and Review [7] require additional convolutions for feature alignments.”
• Anomaly Detection via Reverse Distillation from One-Class Embedding (2022, methodology): “In knowledge distillation (KD) [6, 15], knowledge is transferred within a teacher-student (T-S) pair.”
• Decoupled Knowledge Distillation (2022, positive): “Thus, we introduce the feature-based distillation method ReviewKD [1] to obtain satisfactory results.”