Amalgamating Knowledge from Heterogeneous Graph Neural Networks

By Yongcheng Jing, Yiding Yang, Xinchao Wang, Mingli Song, Dacheng Tao. 2021

105 citations recorded by the source corpus.

In this paper, we study a novel knowledge transfer task in the domain of graph neural networks (GNNs). We strive to train a multi-talented student GNN, without accessing human annotations, that “amalgamates” knowledge from a couple of teacher GNNs with heterogeneous architectures and handling distinct tasks. The student derived in this way is expected to integrate the expertise from both teachers while maintaining a compact architecture. To this end, we propose an innovative approach to train a slimmable GNN that enables learning from teachers with varying feature dimensions. Meanwhile, to explicitly align topological semantics between the student and teachers, we introduce a topological attribution map (TAM) to highlight the structural saliency in a graph, based on which the student imitates the teachers’ ways of aggregating information from neighbors. Experiments on seven datasets across various tasks, including multi-label classification and joint segmentation-classification, demonstrate that the learned student, with a lightweight architecture, achieves gratifying results on par with and sometimes even superior to those of the teachers in their specializations. Our code is publicly available at https://github.com/ycjing/AmalgamateGNN.PyTorch.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xinchao Wang
2. Dacheng Tao
3. Liu Liu
4. Jingwen Ye
5. Yajing Kong
6. Xingyi Yang
7. Gongfan Fang
8. Xinyin Ma

In Computer Vision and Pattern Recognition

1. Xinchao Wang
2. Dacheng Tao
3. Jingwen Ye
4. Xingyi Yang
5. Liu Liu
6. Yajing Kong
7. Gongfan Fang
8. Xinyin Ma

Citations in the corpus

• Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, neutral): “Regularization-based methods can be divided into two classes: one is to distill knowledge from the previous model which is trained on the previous tasks [14, 15, 23, 36, 52, 59]; another is to explicitly use a regularizer to penalize the update of important parameters of previous tasks, preventing t”
• Factorizing Knowledge in Neural Networks (2022, background): “KD is originally designed for model compression [5,56,32,37,64,51,22], but it has been found to be beneficial in other tasks like adversarial defense [47], domain adaptation [15,44], continual learning [33,68] and amalgamate the knowledge from multiple teachers [39,65,24].”
• DeepCache: Accelerating Diffusion Models for Free (2023, background): “Some other works [17, 20, 23] transferred the knowledge into the distilled model.”
• Adaptive Curriculum Learning (2021, background): “KL divergence is frequently used to maintain useful knowledge learned from a pre-trained model [4, 5, 17, 43].”