Relational Knowledge Distillation

By Wonpyo Park, Dongju Kim, Yan Lu, Minsu Cho. 2019

1,966 citations recorded by the source corpus.

Knowledge distillation aims at transferring knowledge acquired in one model (a teacher) to another model (a student) that is typically smaller. Previous approaches can be expressed as a form of training the student to mimic output activations of individual data examples represented by the teacher. We introduce a novel approach, dubbed relational knowledge distillation (RKD), that transfers mutual relations of data examples instead. For concrete realizations of RKD, we propose distance-wise and angle-wise distillation losses that penalize structural differences in relations. Experiments conducted on different tasks show that the proposed method improves educated student models with a significant margin. In particular for metric learning, it allows students to outperform their teachers' performance, achieving the state of the arts on standard benchmark datasets.

Subfields: Artificial Intelligence, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

Citations in the corpus

Contrastive Representation Distillation (2019, methodology): “Additionally, we also consider other distillation objectives, such as KD discussed in previous section, Attention Transfer Zagoruyko & Komodakis (2016a) and FitNet Romero et al. (2014).”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, background): “Relational KD [39] additionally propose to preserve the angle formed by the three examples in the feature space by means of a triplet distillation objective.”
Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning (2022, background): “Various solutions have been proposed to enhance its effectiveness, such as response-based distillation [3], feature-based distillation [8] and relation-based distillation [6].”
Few-Shot Class-Incremental Learning via Relation Knowledge Distillation (2021, background): “(3) Relation-based knowledge further explores the structures between different layers and data samples (Yim et al. 2017; Park et al. 2019; Ramakrishnan et al. 2020).”
SID: Incremental Learning for Anchor-Free Object Detection via Selective and Inter-Related Distillation (2020, methodology): “Inspired by Liu et al. (2019) and Park et al. (2019), we transfer additional information from the source model by considering the relationship among diﬀerent instances.”
Knowledge Distillation: A Survey (2020, methodology): “Park et al. (2019) proposed a relational knowledge distillation, which transfers the knowledge from instance relations.”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, background): “Knowledge distillation and its improved variants [1, 53, 57, 67, 73] enable teaching students with goals such as quantization [42, 55], compact neural network architecture design [57], semantic segmentation [31], self-distillation [14], and un-/semi-supervised learning [34, 54, 70].”
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020, methodology): “Relational Knowledge Distillation ( ) [36] encourages the model to mimic the mutual relations between different samples.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “[94, 148, 187] use logits (the soft probabilities) as the source of knowledge, while [3, 200, 261] try to leverage the knowledge from intermediate layers.”
Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs (2020, background): “Investigating more advanced techniques ( e.g ., [1, 18, 34]) would be an interesting and promising future direction.”
A Survey on Few-Shot Class-Incremental Learning (2023, methodology): “Unlike CIL based on individual knowledge distillation (Park et al., 2019), Dong et al.”
Class-Incremental Learning: A Survey (2023, background): “To reveal the structural information in model distillation, several works suggest conducting relational knowledge distillation [192].”
Complementary Relation Contrastive Distillation (2021, methodology): “To verify its superiority, we compare it with four methods using low-dimensional relations: 1) RKD [32]; 2) CC [34]; 3) SP [43]; and 4) PKT [33].”
Safe Distillation Box (2021, background): “Many recent efforts have focused on improving the efficiency of KD, and showcased that the KD process can be lightened without much compensation on performances (Park et al. 2019; Mirzadeh et al. 2020; Chen et al. 2020).”
Self-supervised Label Augmentation via Input Transformations (2019, neutral): “There are many advanced distillation techniques (Zagoruyko & Komodakis, 2016a; Park et al., 2019; Ahn et al., 2019; Tian et al., 2020), but they should train the larger network first, which leads to high training costs.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “In contrast, Yang et al. [246], Yu et al. [259], Arora et al. [12], RKD [177] and Peng et al. [179] shift to a new perspective focusing more on putting strictness to the distillation process via optimization (e.g., distribution embedding, mutual relations, etc).”
Local Correlation Consistency for Knowledge Distillation (2020, methodology): “Park et al. [18] propose distance-wise and angle-wise distillation losses to penalize structural diﬀerences in relations.”
Matching Guided Distillation (2020, methodology): “Another technique fashion for distillation is to design the loss function, including activation transfer loss with boundaries formed by hidden neurons [12], the loss for penalizing structural differences in relations [26].”
Adaptive multi-teacher multi-level knowledge distillation (2020, methodology): “In addition to the standard knowledge distillation which regards each data example separately, we also consider to add structural knowledge between different data examples to transfer relational information effectively, which is motivated by [17].”