Complementary Relation Contrastive Distillation

By Jinguo Zhu, Shixiang Tang, Dapeng Chen, Shijie Yu, Yakun Liu, Aijun Yang, Mingzhe Rong, Xiaohua Wang. 2021

101 citations recorded by the source corpus.

Knowledge distillation aims to transfer representation ability from a teacher model to a student model. Previous approaches focus on either individual representation distillation or inter-sample similarity preservation. While we argue that the inter-sample relation conveys abundant information and needs to be distilled in a more effective way. In this paper, we propose a novel knowledge distillation method, namely Complementary Relation Contrastive Distillation (CRCD), to transfer the structural knowledge from the teacher to the student. Specifically, we estimate the mutual relation in an anchor-based way and distill the anchor-student relation under the supervision of its corresponding anchor-teacher relation. To make it more robust, mutual relations are modeled by two complementary elements: the feature and its gradient. Furthermore, the low bound of mutual information between the anchor-teacher relation distribution and the anchor-student relation distribution is maximized via relation contrastive loss, which can distill both the sample representation and the inter-sample relations. Experiments on different benchmarks demonstrate the effectiveness of our proposed CRCD.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chen Zhao
2. Jinguo Zhu
3. Shijie Yu
4. Bernard Ghanem
5. Dapeng Chen
6. Shixiang Tang
7. Wanli Ouyang
8. Jian Zhang

In Computer Vision and Pattern Recognition

1. Bernard Ghanem
2. Wanli Ouyang
3. Chen Zhao
4. Dapeng Chen
5. Luc Van Gool
6. Dacheng Tao
7. Xiaogang Wang
8. Shixiang Tang

Citations in the corpus

• Layerwise Optimization by Gradient Decomposition for Continual Learning (2021, positive): “The latter approaches, such as LWF [29], LFL [20], LWM [9] and BLD [11], leveraged knowledge distillation [16, 58] among consecutive tasks.”
• R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learning (2022, methodology): “We also investigated some newer KD methods, such as CRD [27] and CRCD [34].”
• Anomaly Detection via Reverse Distillation from One-Class Embedding (2022, methodology): “For knowledge transfer in the T-S model, the cosine similarity is taken as the KD loss, as it is more precisely to capture the relation in both highand low-dimensional information [37, 49].”