Differentiable Feature Aggregation Search for Knowledge Distillation

By Yushuo Guan, Pengyu Zhao, Bingxuan Wang, Yuanxing Zhang, Cong Yao, Kaigui Bian, Jian Tang. 2020

55 citations recorded by the source corpus.

Knowledge distillation has become increasingly important in model compression. It boosts the performance of a miniaturized student network with the supervision of the output distribution and feature maps from a sophisticated teacher network. Some recent works introduce multi-teacher distillation to provide more supervision to the student network. However, the effectiveness of multi-teacher distillation methods are accompanied by costly computation resources. To tackle with both the efficiency and the effectiveness of knowledge distillation, we introduce the feature aggregation to imitate the multi-teacher distillation in the single-teacher distillation framework by extracting informative supervision from multiple teacher feature maps. Specifically, we introduce DFA, a two-stage Differentiable Feature Aggregation search method that motivated by DARTS in neural architecture search, to efficiently find the aggregations. In the first stage, DFA formulates the searching problem as a bi-level optimization and leverages a novel bridge loss, which consists of a student-to-teacher path and a teacher-to-student path, to find appropriate feature aggregations. The two paths act as two players against each other, trying to optimize the unified architecture parameters to the opposite directions while guaranteeing both expressivity and learnability of the feature aggregation simultaneously. In the second stage, DFA performs knowledge distillation with the derived feature aggregation. Experimental results show that DFA outperforms existing methods on CIFAR-100 and CINIC-10 datasets under various teacher-student settings, verifying the effectiveness and robustness of the design.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jianping Gou
3. Yifan Liu
4. Lan Du
5. Liyuan Sun
6. Chunhua Shen
7. Changyong Shu
8. Defang Chen

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Chunhua Shen
3. Yifan Liu
4. Jianping Gou
5. Luc Van Gool
6. Lan Du
7. Liyuan Sun
8. Xiaogang Wang

Citations in the corpus

• Knowledge Distillation: A Survey (2020, neutral): “DFA (Guan et al. 2020) Feature aggregation Hint layer L2(.”
• Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, methodology): “To overcome the limitation of knowledge from the output layer of teacher, the feature-based knowledge from the middle layers of teacher is also used to train the student [21, 18, 19].”
• Cross-Layer Distillation with Semantic Calibration (2020, background): “The most related approach to ours, which is proposed recently, works on channel or group-wise association within given teacherstudent layer pairs [21], [38].”
• Channel-wise Knowledge Distillation for Dense Prediction* (2020, background): “Most works on knowledge distillation focus on classification tasks [10, 11, 15, 26, 36, 38, 44].”