Distilling Knowledge From Graph Convolutional Networks

By Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, Xinchao Wang. 2020

287 citations recorded by the source corpus.

Existing knowledge distillation methods focus on convolutional neural networks (CNNs), where the input samples like images lie in a grid domain, and have largely overlooked graph convolutional networks (GCN) that handle non-grid data. In this paper, we propose to our best knowledge the first dedicated approach to distilling knowledge from a pre-trained GCN model. To enable the knowledge transfer from the teacher GCN to the student, we propose a local structure preserving module that explicitly accounts for the topological semantics of the teacher. In this module, the local structure information from both the teacher and the student are extracted as distributions, and hence minimizing the distance between these distributions enables topology-aware knowledge transfer from the teacher, yielding a compact yet high-performance student model. Moreover, the proposed approach is readily extendable to dynamic graph models, where the input graphs for the teacher and the student may differ. We evaluate the proposed method on two different datasets using GCN models of different architectures, and demonstrate that our method achieves the state-of-the-art knowledge distillation performance for GCN models.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Graph Learning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Graph Learning

In Efficiency & Compression

Citations in the corpus

Learning with Recoverable Forgetting (2022, positive): “Here, we mainly discuss the related works in knowledge distillation [15,14,54], which trains a student model of a compact size by learning from a larger teacher model or a set of teachers handling the same task.”
Class-Incremental Instance Segmentation via Multi-Teacher Networks (2021, methodology): “Speciﬁcally, we utilize knowledge distillation to simulate the guidance process, which is proposed in (Hinton, Vinyals, and Dean 2015) and widely used in incremental tasks (Li and Hoiem 2017; Hou et al. 2018; Wei, Deng, and Yang 2020) and other tasks (Yang et al. 2020c).”
Knowledge Distillation: A Survey (2020, methodology): “hen et al., 2019b;Furlanello et al.,2018; Yang et al., 2019a; Zhang et al., 2018b;Lee et al., 2019c;Park and Kwak,2019; Papernot et al., 2017;Fukuda et al., 2017; Ruder et al.,2017; Wu et al., 2019a; Yang et al., 2020b; Vongkulbhisal et al., 2019). A generic framework for multi-teacher distillation ”
Progressive Network Grafting for Few-Shot Knowledge Distillation (2020, unknown): “The works of (Song et al. 2019, 2020), on the other hand, focus on estimating knowledge transferability across different tasks, while those of (Yang et al. 2020b,a) explore distillation on the graph domain.”
GraphSAIL: Graph Structure Aware Incremental Learning for Recommender Systems (2020, background): “Knowledge distillation techniques were proposed initially to transfer knowledge from a large and complex model into a smaller distilled model [9, 27].”
Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning (2022, methodology): “Convolution has been used for a long time as the main module to extract the image features in computer vision tasks, and CNN-based architectures have been studied [49, 18, 59, 39, 60, 37, 63] with extension on graph-based data [62, 61, 36].”
Factorizable Graph Convolutional Networks (2020, background): “Graph convolutional network (GCN) has shown its potential in the non-grid domain [Xu et al., 2018, Qiu et al., 2020, Li et al., 2018, Yang et al., 2020b, Monti et al., 2017, Yang et al., 2019], achieving promising results on various type of structural data, like citation graph [Veliˇckovi´c et al.,…”
Factorizing Knowledge in Neural Networks (2022, background): “KD is originally designed for model compression [5,56,32,37,64,51,22], but it has been found to be beneﬁcial in other tasks like adversarial defense [47], domain adaptation [15,44], continual learning [33,68] and amalgamate the knowledge from multiple teachers [39,65,24].”
LLM-Pruner: On the Structural Pruning of Large Language Models (2023, background): “With the growing size of models, there is an increasing demand for efficient LLM compression and compression is independent of the original training data.”
Safe Distillation Box (2021, methodology): “Knowledge distillation is first introduced as a technique for neural network compression (Hinton, Vinyals, and Dean 2015; Yang et al. 2020), which aims at training a student model of a compact size by learning from a larger teacher model.”
Collaboration by Competition: Self-coordinated Knowledge Amalgamation for Multi-talent Student Learning (2020, background): “Knowledge distillation (KD) [8, 9, 29, 35] adopts a teacher-guiding student strategy where a small student network learns to imitate the output of a large and deeper teacher network.”
Amalgamating Knowledge from Heterogeneous Graph Neural Networks (2021, methodology): “The work of [43], as the first attempt, generalizes knowledge distillation to GNNs, and introduces a customized approach tailored for irregular data.”
Overcoming Catastrophic Forgetting in Graph Neural Networks (2020, background): “2018), and thus falls within the domain of model reuse (Yang et al. 2020b; Ye et al. 2019; Shen et al. 2019).”
Vision GNN: An Image is Worth Graph of Nodes (2022, background): “GCN has been explored for classifying and segmenting points clouds [28, 58, 69].”
Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts (2020, methodology): “al [28] is to augment the training data by using the predicted logits/soft-label from the teacher model; the other [31, 32, 33] adds the loss (i.”
How Attentive are Graph Attention Networks? (2021, background): “…definition of Veličković et al. (2018) (Equation (2)) has become the de facto practice: it has spread to a variety of domains (Zhang et al., 2020; Yang et al., 2020; Wang et al., 2019c; Huang and Carley, 2019; Ma et al., 2020; Kosaraju et al., 2019; Nathani et al., 2019; Wu et al., 2020), and…”