Graph-based Knowledge Distillation by Multi-head Attention Network

By Seunghyun Lee, Byung Cheol Song. 2019

89 citations recorded by the source corpus.

Knowledge distillation (KD) is a technique to derive optimal performance from a small student network (SN) by distilling knowledge of a large teacher network (TN) and transferring the distilled knowledge to the small SN. Since a role of convolutional neural network (CNN) in KD is to embed a dataset so as to perform a given task well, it is very important to acquire knowledge that considers intra-data relations. Conventional KD methods have concentrated on distilling knowledge in data units. To our knowledge, any KD methods for distilling information in dataset units have not yet been proposed. Therefore, this paper proposes a novel method that enables distillation of dataset-based knowledge from the TN using an attention network. The knowledge of the embedding procedure of the TN is distilled to graph by multi-head attention (MHA), and multi-task learning is performed to give relational inductive bias to the SN. The MHA can provide clear information about the source dataset, which can greatly improves the performance of the SN. Experimental results show that the proposed method is 7.05% higher than the SN alone for CIFAR100, which is 2.46% higher than the state-of-the-art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Xiaopeng Hong
3. Xing Wei
4. Xiaoyu Tao
5. Yihong Gong
6. Xinyuan Chang
7. Yu Qiao
8. Jianping Gou

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Xiaogang Wang
4. Yu Qiao
5. Ming-Hsuan Yang
6. Trevor Darrell
7. Bernt Schiele
8. Yi Yang

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “Most of knowledge distillation algorithms focus on transferring knowledge from the teacher to the student in terms of each training sample, while several recent methods have been proposed to explore the intradata relationships (Luo et al., 2018; Chen et al., 2020; Zhang and Peng, 2018; Lee and Song,”
• Self-Distillation: Towards Efficient and Compact Neural Networks (2021, methodology): “Then, fruitful distillation methods have been proposed to transfer the knowledge of teacher models to student models based on feature maps [35], attention [36], the flow of solution procedure [37], graph [38] and GAN-based methods [39], [40], [41].”
• Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher (2020, background): “In [12] and [14], the authors consider graph-based distillation.”
• Topology-Preserving Class-Incremental Learning (2020, methodology): “Most class-incremental learning (CIL) works [32, 3, 42, 13] alleviates forgetting using the knowledge distillation [12, 43, 31, 18] technique, which is initially introduced by LWF [19] for multi-task incremental learning.”
• Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition (2020, methodology): “A graph is generated by the Multi-Head Attention (MHA) (Seunghyun Lee 2019) which provides a relational inductive bias to improve the performance significantly.”