Training Graph Neural Networks with 1000 Layers

By Guohao Li, Matthias Müller, Bernard Ghanem, Vladlen Koltun. International Conference on Machine Learning, 2021

302 citations recorded by the source corpus.

Deep graph neural networks (GNNs) have achieved excellent results on various tasks on increasingly large graph datasets with millions of nodes and edges. However, memory complexity has become a major obstacle when training deep GNNs for practical applications due to the immense number of nodes, edges, and intermediate activations. To improve the scalability of GNNs, prior works propose smart graph sampling or partitioning strategies to train GNNs with a smaller set of nodes or sub-graphs. In this work, we study reversible connections, group convolutions, weight tying, and equilibrium models to advance the memory and parameter efficiency of GNNs. We find that reversible connections in combination with deep network architectures enable the training of overparameterized GNNs that significantly outperform existing methods on multiple datasets. Our models RevGNN-Deep (1001 layers with 80 channels each) and RevGNN-Wide (448 layers with 224 channels each) were both trained on a single commodity GPU and achieve an ROC-AUC of $87.74 \pm 0.13$ and $88.24 \pm 0.15$ on the ogbn-proteins dataset. To the best of our knowledge, RevGNN-Deep is the deepest GNN in the literature by one order of magnitude. Please visit our project website https://www.deepgcns.org/arch/gnn1000 for more information.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/cf30fb61a5943781144c8442563e3ef9c38df871

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chunyan Miao
2. Shuyuan Xu
3. Ruosong Ye
4. Xin Zhou
5. Hongyu Zhou
6. Yuan You
7. Yongfeng Zhang
8. Caiqi Zhang

Citations in the corpus

• GraphGPT: Graph Instruction Tuning for Large Language Models (2023, unknown): “Each paper in the dataset is associated with a research category, manually labeled by the authors and arXiv moderators.”
• Language is All a Graph Needs (2023, background): “…self-attention (Vaswani et al., 2017) applied on such inputs can be seen as an advanced multi-scale weighted average aggregation mechanism of GATs (Veliˇckovi´c et al., 2017; Li et al., 2021), facilitating InstructGLM to effectively grasp different neighbors’ varying importance to the central node.”
• Bootstrap Latent Representations for Multi-modal Recommendation (2022, background): “GCNs may sufer from the over-smoothing problem [2, 16, 18].”
• PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies (2022, unknown): “Effectively expanding PointNet++ from its original small scale to a larger scale is a topic worth studying because larger models are generally expected to enable richer representations and perform better [2, 18, 61].”
• Exploring the Potential of Large Language Models (LLMs)in Learning on Graphs (2023, unknown): “We first study feature-level enhancement .”