Matching Guided Distillation

By Kaiyu Yue, Jiangfan Deng, Feng Zhou. 2020

63 citations recorded by the source corpus.

Feature distillation is an effective way to improve the performance for a smaller student model, which has fewer parameters and lower computation cost compared to the larger teacher model. Unfortunately, there is a common obstacle - the gap in semantic feature structure between the intermediate features of teacher and student. The classic scheme prefers to transform intermediate features by adding the adaptation module, such as naive convolutional, attention-based or more complicated one. However, this introduces two problems: a) The adaptation module brings more parameters into training. b) The adaptation module with random initialization or special transformation isn't friendly for distilling a pre-trained student. In this paper, we present Matching Guided Distillation (MGD) as an efficient and parameter-free manner to solve these problems. The key idea of MGD is to pose matching the teacher channels with students' as an assignment problem. We compare three solutions of the assignment problem to reduce channels from teacher features with partial distillation loss. The overall training takes a coordinate-descent approach between two optimization objects - assignments update and parameters update. Since MGD only contains normalization or pooling operations with negligible computation cost, it is flexible to plug into network with other distillation methods.

Subfields: Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Changyong Shu
2. Yifan Liu
3. Chunhua Shen
4. Jianfei Gao
5. Defang Chen
6. Zheng Yan
7. Jian-Ping Mei
8. Chixiang Zhang

Citations in the corpus

• Cross-Layer Distillation with Semantic Calibration (2020, methodology): “The most related approach to ours, which is proposed recently, works on channel or group-wise association within given teacherstudent layer pairs [21], [38].”
• Channel-wise Knowledge Distillation for Dense Prediction* (2020, background): “Other works, such as MGD [40], Channel exchanging [33] and CSC [25] show the importance of channel-wise information.”
• Learning Compatible Embeddings (2021, background): “Various knowledge sources are adopted in literature such as classification logits [17, 1, 28], hint layers [35, 54, 49, 3] and multi-layer groups [52, 51, 20].”