GPT-GNN: Generative Pre-Training of Graph Neural Networks

By Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, Yizhou Sun. Knowledge Discovery and Data Mining, 2020

687 citations recorded by the source corpus.

Graph neural networks (GNNs) have been demonstrated to be powerful in modeling graph-structured data. However, training GNNs requires abundant task-specific labeled data, which is often arduously expensive to obtain. One effective way to reduce the labeling effort is to pre-train an expressive GNN model on unlabelled data with self-supervision and then transfer the learned model to downstream tasks with only a few labels. In this paper, we present the GPT-GNN framework to initialize GNNs by generative pre-training. GPT-GNN introduces a self-supervised attributed graph generation task to pre-train a GNN so that it can capture the structural and semantic properties of the graph. We factorize the likelihood of graph generation into two components: 1) attribute generation and 2) edge generation. By modeling both components, GPT-GNN captures the inherent dependency between node attributes and graph structure during the generative process. Comprehensive experiments on the billion-scale open academic graph and Amazon recommendation data demonstrate that GPT-GNN significantly outperforms state-of-the-art GNN models without pre-training by up to 9.1% across various downstream tasks?

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/04faf433934486c41d082e8d75ccfe5dc2f69fef

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jie Tang
3. Philip S. Yu
4. Shirui Pan
5. Yuxiao Dong
6. Jiliang Tang
7. Hongzhi Yin
8. Hongxia Yang

In Information Systems

1. Hongzhi Yin
2. Junliang Yu
3. Tong Chen
4. Xin Xia
5. Dacheng Tao
6. Zi-Liang Huang
7. Jie Tang
8. Jiliang Tang

Citations in the corpus

• A Survey on Heterogeneous Graph Embedding: Methods, Techniques, Applications and Sources (2020, neutral): “Another important direction is the pre-training of HGNNs [148], [149], [150].”
• Exploring the Potential of Large Language Models (LLMs)in Learning on Graphs (2023, result): “According to the results in Table 10, it is evident that LLMs demonstrate remarkable zero-shot performance on Pubmed .”
• Self-Supervised Learning for Recommender Systems: A Survey (2022, methodology): “Inspired by GPT-GNN [120], PMGT [117] also sets up a feature reconstruction task to pre-tain the Transformer-based recommendation model.”
• SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation (2022, background): “GPT-GNN [16] designs generative task in which node attributes and edges are alternatively generated such that the likelihood of a graph is maximized.”
• Neural, symbolic and neural-symbolic reasoning on knowledge graphs (2021, background): “Recently, the transferable pre-training graph neural networks have proved to be able to capture the graph structure characteristics across different graph data (Hu et al., 2020; Qiu et al., 2020a).”
• Graph Contrastive Learning with Augmentations (2020, background): “If extended to pre-training GCNs, this framework can potentially overcome the aforementioned limitations of proximity-based pre-training methods [22, 23, 34, 35, 36, 37, 38, 39].”
• Graph Self-Supervised Learning: A Survey (2021, methodology): “GPT-GNN [9] integrates feature and structure generation into a pre-training”
• OAG-BERT: Pre-train Heterogeneous Entity-augmented Academic Language Models (2021, methodology): “Recent works including Heterogeneous Graph Transformer [17] and GPT-GNN [16] similarly borrow the idea from the natural language community, applying transformer blocks and pre-training scheme on graph tasks.”
• Pre-training Molecular Graph Representation with 3D Geometry (2021, methodology): “Model 2D Topology 3D Geometry EdgePred [31] X AttrMask [38] X GPT-GNN [39] X InfoGraph [71, 79] X ContexPred [38] X GraphLoG [88] X GraphCL [91] X JOAO [90] X Grover [66] X GraphMVP (Ours) X X”
• A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “…with clustering [68], semi-supervised learning [145], multi-task learning [157], [158], transfer learning [159], [160], [161], graph NNs [147], [162], [163], reinforcement learning [164], [165], [166], few-shot learning [167], [168], neural architecture search [169], robust learning [146],…”
• Are we really making much progress?: Revisiting, benchmarking and refining heterogeneous graph neural networks (2021, background): “As graph neural networks (GNNs) [2, 21] have already occupied the centre stage of graph mining research within recent years, the researchers begin to pay attention to their potential on heterogeneous graphs (a.k.a., Heterogeneous Information Networks) [8, 12, 19, 36, 40, 43].”
• Self-Supervised Learning on Graphs: Contrastive, Generative, or Predictive (2021, methodology): “Moreover, due to their simplicity and effectiveness, thesemethods have been widely used in algorithms such as GPT-GNN [32] and recommendation applications such as Pretrain-Recsys”
• Self-Supervised Representation Learning: Introduction, advances, and challenges (2021, neutral): “A minority of methods are inductive [27], [57] in that they learn embedding functions that do not depend on a specific choice of input graph, and thus can be transferred to new target nodes or graphs.”
• Unsupervised Point Cloud Pre-training via Occlusion Completion (2020, background): “(3) Mask-based completion tasks have become the de facto standard for learning pre-trained representations in natural language processing [11, 32, 36] and are widely used in pre-training for images [35] and graphs [24].”
• Self-Supervised Learning: Generative or Contrastive (2020, background): “GPT-GNN’s wide range of experiments on OAG [116], [124], [154], the largest public, academic graph with 100 million nodes and 2 billion edges, shows impressive improvements on various graph learning tasks.”