Strategies for Pre-training Graph Neural Networks

By Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, Jure Leskovec. International Conference on Learning Representations, 2019

1,844 citations recorded by the source corpus.

Many applications of machine learning require a model to make accurate pre-dictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training. An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and then fine-tune it on a downstream task of interest. While pre-training has been effective in many language and vision domains, it remains an open question how to effectively use pre-training on graph datasets. In this paper, we develop a new strategy and self-supervised methods for pre-training Graph Neural Networks (GNNs). The key to the success of our strategy is to pre-train an expressive GNN at the level of individual nodes as well as entire graphs so that the GNN can learn useful local and global representations simultaneously. We systematically study pre-training on multiple graph classification datasets. We find that naive strategies, which pre-train GNNs at the level of either entire graphs or individual nodes, give limited improvement and can even lead to negative transfer on many downstream tasks. In contrast, our strategy avoids negative transfer and improves generalization significantly across downstream tasks, leading up to 9.4% absolute improvements in ROC-AUC over non-pre-trained models and achieving state-of-the-art performance for molecular property prediction and protein function prediction.

Subfields: Artificial Intelligence, Materials Chemistry, Representation & Self-Supervised Learning, Graph Learning

View Paper: https://arxiv.org/abs/1905.12265

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

In Representation & Self-Supervised Learning

In Graph Learning

Citations in the corpus

Learning Gradient Fields for Molecular Conformation Generation (2021, methodology): “…and the edge attributes into low dimensional feature spaces using feedforward networks: h0i = MLP(Zi), ∀vi ∈ V, heij = MLP(eij , dij), ∀eij ∈ E. (4) We then adopt a variant of Graph Isomorphism Network (GINs) (Xu et al., 2019; Hu et al., 2020) to compute node embeddings based on graph structures.”
Self-Supervised Representation Learning: Introduction, advances, and challenges (2021, neutral): “Depending on whether the ultimate task of interest requires node-or graph-level predictions, methods may focus on learning node-[15], [27], [57] or graph-level [90] representations, or both [91].”
Self-Supervised Learning on Graphs: Contrastive, Generative, or Predictive (2021, methodology): “The focus on three different levels of nodes, edges, and structures has led to different generative methods such as Graph Completion [12], Edge Feature Masking [11], and Ad-jacency Matrix Reconstruction [14], respectively.”
ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction (2020, methodology): “In particular, the masked language-modeling (MLM) pretraining task [8] commonly used for BERT-style architectures is analogous to atom masking tasks used in graph settings [14].”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, methodology): “It has been combined with clustering [66], semi-supervised learning [151], multi-task learning [163]–[166], transfer learning [167]–[169], graph NNs [153], [170]–[175], reinforcement learning [176]–[178], few-shot learning [179], [180], neural”
Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development (2021, methodology): “In addition, recently, [53] has adapted a pre149 training strategy to molecule graph, where we include two strategies attribute masking (AttMasking) 150 and context prediction (ContextPred).”
Do Transformers Really Perform Badly for Graph Representation? (2021, neutral): “…± 0.30 GINE-VN [5] 6.1M 29.17 ± 0.15 PHC-GNN [29] 1.7M 29.47 ± 0.26 GINE-APPNP [5] 6.1M 29.79 ± 0.30 GIN-VN [49] (fine-tune) 3.4M 29.02 ± 0.17 Graphormer-FLAG 119.5M 31.39 ± 0.32 [49] the other pre-trained GNNs do not achieve competitive performance, which is in line with previous literature [20].”
Do Transformers Really Perform Bad for Graph Representation? (2021, result): “the other pre-trained GNNs do not achieve competitive performance, which is in line with previous literature [19].”
TorchDrug: A Powerful and Flexible Machine Learning Platform for Drug Discovery (2022, methodology): “(Hu et al., 2019) De Novo Molecule Design and Optimization GCPN (You et al., 2018) GraphAF (Shi et al., 2020b) Retrosynthesis Prediction G2Gs (Shi et al., 2020a)”
Self-supervised Graph Learning for Recommendation (2020, background): “Self-supervised learning has proved its superiority in pre-training natural language model [8] and graph structure [21, 31].”
How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks (2020, methodology): “, BERT) with unlabeled out-of-distribution data in X \ D (Chen et al., 2020; Devlin et al., 2019; Hu et al., 2020; Mikolov et al., 2013b; Peters et al., 2018).”
Geometry-enhanced molecular representation learning for property prediction (2021, methodology): “Recently, some 316 studies [23, 40] apply self-supervised learning methods to GNNs for molecular property prediction 317 to overcome the insufficiency of the labeled samples.”
Towards Domain-Agnostic Contrastive Learning (2020, background): “For graph-structured data, the pretext task can be predicting the context (neighbourhood of a given node) or predicting the masked attributes of the node (Hu et al., 2020).”