Correlation Congruence for Knowledge Distillation

By Baoyun Peng, Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Yu Liu, Dongsheng Li, Zhaoning Zhang. 2019

645 citations recorded by the source corpus.

Most teacher-student frameworks based on knowledge distillation (KD) depend on a strong congruent constraint on instance level. However, they usually ignore the correlation between multiple instances, which is also valuable for knowledge transfer. In this work, we propose a new framework named correlation congruence for knowledge distillation (CCKD), which transfers not only the instance-level information but also the correlation between instances. Furthermore, a generalized kernel method based on Taylor series expansion is proposed to better capture the correlation between instances. Empirical experiments and ablation studies on image classification tasks (including CIFAR-100, ImageNet-1K) and metric learning tasks (including ReID and Face Recognition) show that the proposed CCKD substantially outperforms the original KD and other SOTA KD-based methods. The CCKD can be easily deployed in the majority of the teacher-student framework such as KD and hint-based learning methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Contrastive Representation Distillation (2019, methodology): “We thank Baoyun Peng for providing the code of CC Peng et al. (2019) and Frederick Tung for verifying our reimplementation of SP Tung & Mori (2019). This research was supported in part by Google Cloud and iFlytek.”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, background): “Extending this direction, Correlation Congruence [40] models relations between examples through higherorder kernels, to enforce the same objectives with better relation estimates.”
Knowledge Distillation: A Survey (2020, methodology): “Recently, knowledge distillation has been used successfully for solving the complex image classiﬁcation problems (Zhu et al. 2019; Bagherinezhad et al. 2018; Peng et al. 2019b; Li and Hoiem 2017; Chen et al. 2018a; Wang et al. 2019b; Mukherjee et al. 2019).”
Patch-based Knowledge Distillation for Lifelong Person Re-Identification (2022, positive): “Existing knowledge distillation methods can be summarized into three categories: logit distillation [3, 21] which matches the final predictions, feature distillation [49, 67] which matches the intermediate representations, and relation distillation [37, 42, 45, 60] which matches inter-sample relatio”
Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation (2021, methodology): “The intermediate layer distillation methods were then introduced, so such methods utilize the teacher network’s knowledge from either the convolutional layer with feature-map level preserving localities [26, 36, 14, 34, 16]; or penultimate layer [24, 29, 21, 23, 30].”
Complementary Relation Contrastive Distillation (2021, methodology): “To verify its superiority, we compare it with four methods using low-dimensional relations: 1) RKD [32]; 2) CC [34]; 3) SP [43]; and 4) PKT [33].”
Cross-Layer Distillation with Semantic Calibration (2020, background): “Therefore, a variety of knowledge distillation approaches have been proposed based on feature embedding, especially the generated relational graphs where each node represents one instance (Passalis and Tefas 2018; Peng et al. 2019; Park et al. 2019; Liu et al. 2019).”
Prime-Aware Adaptive Distillation (2020, methodology): “ns. For example, PAD obtains a 1.71% gain for FitNet on CIFAR100 and 2.56% for L 2 on TinyImagenet. PAD combined L 2 outperforms other methods. We also nd that relation-level distillation methods (CC [25] and RKD [24]) are not superior to the instance-level methods (HKD [13], FitNet [28], and L 2) o”
ISD: Self-Supervised Learning by Iterative Similarity Distillation (2020, methodology): “Similarity based knowledge distillation: While the above methods [16, 3, 30, 36, 38] only extract the information about a single data point from the teacher, similarity based distillation methods [27, 29, 26, 33, 2, 32, 1] represent the knowledge from the teacher in terms of similarities between dat”
Inter-Region Affinity Distillation for Road Marking Segmentation (2020, neutral): “There exist previous distillation methods [15, 17, 23] that exploit the relationship between a batch of samples.”
Knowledge Distillation from A Stronger Teacher (2022, background): “Besides this vanilla prediction match, other works [11, 14, 34, 41] also investigate the knowledge within intermediate representations to further boost the distillation performance, but this usually induces additional training cost as a consequence.”
Search to Distill: Pearls Are Everywhere but Not the Eyes (2019, methodology): “It is much harder to learn a complex data distribution for a tiny neural network, but some previous works [13, 22] have shown that a huge improvement can be achieved by introducing KD in complex metric learning.”
Relationship Matters: Relation Guided Knowledge Transfer for Incremental Learning of Object Detectors (2020, positive): “With growing interests in knowledge distillation, task-specific KD methods have also been proposed for object detection [39, 4, 40, 29] but with fixed number of classes.”
Local Correlation Consistency for Knowledge Distillation (2020, methodology): “Both Tung et al. [27] and Peng et al. [20] hope to preserve the pair-wise similarity based on the correlation consistency.”
Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, background): “Peng et al. [31] designed a knowledge transfer method that makes sure the student matches the instance correlation consistently with the teacher.”
Curriculum Temperature for Knowledge Distillation (2022, neutral): “2021) and relationshipbased (Park et al. 2019; Peng et al. 2019) methods.”
Decoupled Knowledge Distillation (2022, background): “The following works can be divided into two types, distillation from logits [3,6,22,40,44] and intermediate features [10, 11, 14, 15, 23, 25, 28, 33, 34, 41, 43].”
Subclass Distillation (2020, methodology): “Several distillation methods have been proposed in the last few years (Zagoruyko & Komodakis, 2016; Tung & Mori, 2019; Peng et al., 2019; Ahn et al., 2019; Park et al., 2019; Passalis & Tefas, 2018; Heo et al., 2018; Kim et al., 2018; Yim et al., 2017; Huang & Wang, 2017).”
Exclusivity-Consistency Regularized Knowledge Distillation for Face Recognition (2020, background): “Peng et al. [35] use the knowledge of probability consistency to transfer not only the instance-level information, but also the correlation be-tween instances.”
Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space (2020, methodology): “For models trained on ImageNet, we follow the setting in [21].”