Graph Representation Learning via Multi-task Knowledge Distillation

By Jiaqi W. Ma, Qiaozhu Mei. 2019

19 citations recorded by the source corpus.

Machine learning on graph structured data has attracted much research interest due to its ubiquity in real world data. However, how to efficiently represent graph data in a general way is still an open problem. Traditional methods use handcraft graph features in a tabular form but suffer from the defects of domain expertise requirement and information loss. Graph representation learning overcomes these defects by automatically learning the continuous representations from graph structures, but they require abundant training labels, which are often hard to fulfill for graph-level prediction problems. In this work, we demonstrate that, if available, the domain expertise used for designing handcraft graph features can improve the graph-level representation learning when training labels are scarce. Specifically, we proposed a multi-task knowledge distillation method. By incorporating network-theory-based graph metrics as auxiliary tasks, we show on both synthetic and real datasets that the proposed multi-task learning method can improve the prediction performance of the original learning task, especially when the training data size is small.

Subfields: Artificial Intelligence, Information Systems, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Xialei Liu
3. Wei-Hong Li
4. Jianping Gou
5. Hakan Bilen
6. S. Maybank
7. Baosheng Yu
8. Lin Wang

In Information Systems

1. Dacheng Tao
2. S. Maybank
3. Lin Wang
4. Kuk-Jin Yoon
5. Charu C. Aggarwal

In Molecular Biology

1. Dacheng Tao
2. S. Maybank
3. Lin Wang
4. Hakan Bilen
5. Kuk-Jin Yoon

Citations in the corpus

• Knowledge Distillation: A Survey (2020, background): “Most of knowledge distillation algorithms focus on transferring individual instance knowledge from the teacher to the student, while some recent methods have been proposed to explore the intra-data relationships using graphs (Chen et al. 2021; Zhang and Peng 2018; Lee and Song 2019; Park et al. 2019”
• Universal Representation Learning from Multiple Domains for Few-shot Classification (2021, background): “Our work is related to knowledge distillation (KD) methods [17, 27, 30, 40, 45, 50] that dis-”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Although [131] consider the instance features and instance relationships as instance graph, and [150] build input graph representation for multi-task knowledge distillation.”