Knowledge Distillation via Instance Relationship Graph

By Yufan Liu, Jiajiong Cao, Bing Li, Chunfen Yuan, Weiming Hu, Yangxi Li, Yun-feng Duan. 2019

329 citations recorded by the source corpus.

The key challenge of knowledge distillation is to extract general, moderate and sufficient knowledge from a teacher network to guide a student network. In this paper, a novel Instance Relationship Graph (IRG) is proposed for knowledge distillation. It models three kinds of knowledge, including instance features, instance relationships and feature space transformation, while the latter two kinds of knowledge are neglected by previous methods. Firstly, the IRG is constructed to model the distilled knowledge of one network layer, by considering instance features and instance relationships as vertexes and edges respectively. Secondly, an IRG transformation is proposed to models the feature space transformation across layers. It is more moderate than directly mimicking the features at intermediate layers. Finally, hint loss functions are designed to force a student's IRGs to mimic the structures of a teacher's IRGs. The proposed method effectively captures the knowledge along the whole network via IRGs, and thus shows stable convergence and strong robustness to different network architectures. In addition, the proposed method shows superior performance over existing methods on datasets of various scales.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

SID: Incremental Learning for Anchor-Free Object Detection via Selective and Inter-Related Distillation (2020, methodology): “Inspired by Liu et al. (2019) and Park et al. (2019), we transfer additional information from the source model by considering the relationship among diﬀerent instances.”
Knowledge Distillation: A Survey (2020, methodology): “Furthermore, the parameters of the teacher model (or the connections between layers) also contain another knowledge (Liu et al. 2019c).”
Cross-Layer Distillation with Semantic Calibration (2020, background): “Therefore, a variety of knowledge distillation approaches have been proposed based on feature embedding, especially the generated relational graphs where each node represents one instance (Passalis and Tefas 2018; Peng et al. 2019; Park et al. 2019; Liu et al. 2019).”
Knowledge Transfer Graph for Deep Collaborative Learning (2019, background): “[22, 31, 19] transfer mutual relations of data samples in a mini-batch.”
Interactive Knowledge Distillation (2020, background): “To enable the student to acquire more meaning information, a lot of studies defined knowledge based on relations rather than individual feature maps [20,18,15,25].”
Knowledge Distillation Meets Self-Supervision (2020, methodology): “features into \factors" and then use a translator to extract \factors" in student network. AB [15] forces student to learn the binarized values of pre-activation map in teacher network. IRG [25] explores whether the similarity between samples transfer more knowledge. KDSVD [19] calls its m”
Balanced Knowledge Distillation for Long-tailed Learning (2021, background): “Apart from the final prediction [35], other types of knowledge, like the intermediate representations [41, 24, 15] or the relationships between different layers [52] or data samples [54, 37, 32], can also be used to guide the learning of the student model.”
General Instance Distillation for Object Detection (2021, unknown): “At present, the typical forms of knowledge can be divided into three categories [7], response-based knowledge [10, 22], feature-based knowledge [27, 35, 9] and relation-based knowledge [22, 20, 31, 33, 15].”
Knowledge distillation via adaptive instance normalization (2020, background): “In [27], the instance feature and relationships are considered as vertexes and edges respectively in a graph and instance relationship graph is proposed to model the feature space transformation across layers.”
Source-Free Domain Adaptation for Semantic Segmentation (2021, background): “Since then, a variety of KD methods has been presented for model compression [28, 2, 50, 32], domain adaptation [52, 53], and multi-modal learning [51, 14, 10].”
Inter-Region Affinity Distillation for Road Marking Segmentation (2020, unknown): “ERFNet-IRG [12] 70.”
Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation (2021, methodology): “The intermediate layer distillation methods were then introduced, so such methods utilize the teacher network’s knowledge from either the convolutional layer with feature-map level preserving localities [26, 36, 14, 34, 16]; or penultimate layer [24, 29, 21, 23, 30].”
Patch-based Knowledge Distillation for Lifelong Person Re-Identification (2022, unknown): “Existing knowledge distillation methods can be summarized into three categories: logit distillation [3, 21] which matches the final predictions, feature distillation [49, 67] which matches the intermediate representations, and relation distillation [37, 42, 45, 60] which matches inter-sample relatio”
Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, background): “In recent years, there are some newly proposed knowledge distillation methods based on structural relations of data samples [26, 30, 31, 6, 5, 4].”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “On the other aspect, the transformation TF s of the guided features (namely, student transform) of the student is also an important step for effective KD. Interestingly, the SOTA works such as AT [260], MINILM [226], FSP [254], Jacobian [202], FT [106], SVD [119], SP [216], KP [268], IRG [131], RCO ”
Show, Attend and Distill: Knowledge Distillation via Attention-based Feature Matching (2021, unknown): “Relational knowledge distillation aims to transfer relation knowledge between data instances (Park et al. 2019; Tung and Mori 2019; Peng et al. 2019; Liu et al. 2019).”
Local Correlation Consistency for Knowledge Distillation (2020, methodology): “While the key challenge of knowledge distillation lies in extracting moderate and suﬃcient knowledge for guidance [16], our method can better guide the learning of the student network.”