DeeperGCN: All You Need to Train Deeper GCNs

By Guohao Li, Chenxin Xiong, Ali K. Thabet, Bernard Ghanem. 2020

529 citations recorded by the source corpus.

Graph Convolutional Networks (GCNs) have been drawing significant attention with the power of representation learning on graphs. Unlike Convolutional Neural Networks (CNNs), which are able to take advantage of stacking very deep layers, GCNs suffer from vanishing gradient, over-smoothing and over-fitting issues when going deeper. These challenges limit the representation power of GCNs on large-scale graphs. This paper proposes DeeperGCN that is capable of successfully and reliably training very deep GCNs. We define differentiable generalized aggregation functions to unify different message aggregation operations (e.g. mean, max). We also propose a novel normalization layer namely MsgNorm and a pre-activation version of residual connections for GCNs. Extensive experiments on Open Graph Benchmark (OGB) show DeeperGCN significantly boosts performance over the state-of-the-art on the large scale graph learning tasks of node property prediction and graph property prediction. Please visit this https URL for more information.

Subfields: Artificial Intelligence, Neurology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jie Tang
2. Shuicheng Yan
3. Bernard Ghanem
4. Tie-Yan Liu
5. Ming Ding
6. Matthias Müller
7. Qiang Liu
8. Guohao Li

In Neurology

1. Shuicheng Yan
2. Ming Ding
3. Pan Zhou
4. Bernard Ghanem
5. Zhouchen Lin
6. Guohao Li

Citations in the corpus

• Masked Label Prediction: Unified Massage Passing Model for Semi-Supervised Classification (2020, methodology): “Specifi-cally, we gain 82.56% ACC in ogbn-products , 86.42% ROC-AUC in ogbn-proteins , which achieves about 0.6-1.6% absolute improvements compared to the newly SOTA methods like DeeperGCN [ Li et al. , 2020 ] .”
• Questions for Flat-Minima Optimization of Modern Neural Networks (2022, methodology): “For each task, we use two of the following GNN architectures and matrix factorization methods: GCN (Kipf & Welling, 2017), DeeperGCN (DGCN) (Li et al., 2020), SAGE (Hamilton et al.”
• Structure-Aware Transformer for Graph Representation Learning (2022, methodology): “We first train a two-layer vanilla Transformer model using RWPE.”
• Do Transformers Really Perform Bad for Graph Representation? (2021, methodology): “In addition, we compare to GIN’s multi-hop variant [5], and 12-layer deep graph network DeeperGCN [29], which also show promising performance on other leaderboards.”
• Graph Neural Networks with Heterophily (2020, methodology): “…of semi-supervised learning (SSL) or collective classification (Sen et al. 2008; McDow-ell, Gupta, and Aha 2007; Rossi et al. 2012) can be solved with iterative methods (J. Neville 2000; Lu and Getoor 2003), graph-based regularization and probabilistic graphical models (London and Getoor 2014).”
• Do Transformers Really Perform Badly for Graph Representation? (2021, methodology): “[26] 2.0M 0.1318 0.1691 (0.1684*) GIN [49] 3.8M 0.1203 0.1537 (0.1536*) GCN-VN [26, 15] 4.9M 0.1225 0.1485 (0.1510*) GIN-VN [49, 15] 6.7M 0.1150 0.1395 (0.1396*) GINE-VN [5, 15] 13.2M 0.1248 0.1430 DeeperGCN-VN [30, 15] 25.5M 0.1059 0.1398 GT [13] 0.6M 0.0944 0.1400 GT-Wide [13] 83.”
• Are we really making much progress?: Revisiting, benchmarking and refining heterogeneous graph neural networks (2021, background): “Recent study [22] finds that well-designed pre-activation implementation could make residual connection great again in GNNs. Node Residual.”
• Training Graph Neural Networks with 1000 Layers (2021, methodology): “number of layers for ResGNN (Li et al., 2020) and RevGNN with full-batch and mini-batch training on the ogbn-products.”
• Language is All a Graph Needs (2023, background): “…from the OGB Leaderboard 1 , including DRGAT, RevGAT, etc., as the baselines (Zhang et al., 2022a; Hamilton et al., 2017; Kipf and Welling, 2016; Li et al., 2020; Han et al., 2023a; Shi et al., 2020; Yu et al., 2022a; Veliˇckovi´c et al., 2017; Sun et al., 2020; Li et al., 2021; Zhang et al.,…”
• Artificial Neural Networks and Machine Learning – ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14–17, 2021, Proceedings, Part III (2021, methodology): “# params MAE ↓ # params DGN 79.70± 0.97 114K 28.85± 0.30 6,732K 0.168± 0.003 98K PNA 79.05± 1.32 326K 28.38± 0.35 6,550K 0.188± 0.004 95K GIN 77.07± 1.49∗ 3,336K 27.03± 0.23∗ 3,374K 0.387± 0.015 103K GCN 76.06± 0.97 537K 24.24± 0.34∗ 2,017K 0.459± 0.006 103K DeeperGCN 78.58± 1.17 532K 27.81± 0.38∗ 5”
• Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models (2022, methodology): “Consistent with the settings used in DeepGCN [96], [97], our experiments were conducted on the ogbn-proteins dataset, optimizing the node feature prediction task at the level of the optimizer.”