Towards Deeper Graph Neural Networks

By Meng Liu, Hongyang Gao, Shuiwang Ji. 2020

738 citations recorded by the source corpus.

Graph neural networks have shown significant success in the field of graph representation learning. Graph convolutions perform neighborhood aggregation and represent one of the most important graph operations. Nevertheless, one layer of these neighborhood aggregation methods only consider immediate neighbors, and the performance decreases when going deeper to enable larger receptive fields. Several recent studies attribute this performance deterioration to the over-smoothing issue, which states that repeated propagation makes node representations of different classes indistinguishable. In this work, we study this observation systematically and develop new insights towards deeper graph neural networks. First, we provide a systematical analysis on this issue and argue that the key factor compromising the performance significantly is the entanglement of representation transformation and propagation in current graph convolution operations. After decoupling these two operations, deeper graph neural networks can be used to learn graph node representations from larger receptive fields. We further provide a theoretical analysis of the above observation when building very deep models, which can serve as a rigorous and gentle description of the over-smoothing issue. Based on our theoretical and empirical analysis, we propose Deep Adaptive Graph Neural Network (DAGNN) to adaptively incorporate information from large receptive fields. A set of experiments on citation, co-authorship, and co-purchase datasets have confirmed our analysis and insights and demonstrated the superiority of our proposed methods.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Training Graph Neural Networks with 1000 Layers (2021, methodology): “Another approach is efficient propagation via K-power adjacency matrices or graph diffusion matrices (Wu et al., 2019; Klicpera et al., 2019; Bojchevski et al., 2020; Liu et al., 2020; Frasca et al., 2020).”
Explainability in Graph Neural Networks: A Taxonomic Survey (2020, background): “Several graphrelated tasks are widely studied, such as node classification [30], [31], [32], graph classification [5], [33], link predictions [34], [35], [36].”
Graph Contrastive Learning Automated (2021, methodology): “Graph neural networks (GNNs) have grown into powerful tools to model non-Euclidean graph-structured data arising from various fields (Xu et al., 2018; You et al., 2020c; You & Shen, 2020; Liu et al., 2020; Zhang et al., 2020).”
Self-Supervised Learning of Graph Neural Networks: A Unified Review (2021, background): “With the rapid development of graph neural networks (GNNs) [23], [24], [25], [26], [27], [28], [29], basic components of GNNs [30], Yaochen Xie, Zhao Xu, Jingtun Zhang, and Shuiwang Ji are with the Department of Computer Science and Engineering, Texas A&M University, College Station, TX 77843 USA.”
A Survey on Oversmoothing in Graph Neural Networks (2023, neutral): “Examples of such models include Jumping Knowledge Networks (JKNets) (Xu et al., 2018) and Deep Adaptive Graph Neural Networks (DAGNNs) (Liu et al., 2020)”
Graph Contrastive Learning with Augmentations (2020, background): “Numerous variants of GNNs have been proposed to achieve state-of-the-art performances in graph-based tasks, such as node or link classification [1, 2, 4, 5, 6], link prediction [7] and graph classification [8, 3].”
Motif-based Graph Self-Supervised Learning for Molecular Property Prediction (2021, methodology): “In Table 2 we show that MGSSL is agnostic to the GNN architectures by trying five popular GNN models including GIN [46], GCN [19], RGCN[33], GraphSAGE[11] and DAGNN [24].”
Bootstrap Latent Representations for Multi-modal Recommendation (2022, background): “However, GCNs may suffer from the over-smoothing problem [33]–[35].”
BernNet: Learning Arbitrary Graph Spectral Filters via Bernstein Approximation (2021, background): “[41] further show that GCN [13], DAGNN [19], and JKNet [36] can be interpreted by the optimization function (4) with γ(L) = L.”
Graph Neural Networks in Recommender Systems: A Survey (2020, background): “There are only a few emerging works to adaptively decide the propagation step for each node in order to obtain a reasonable reception field [71, 96, 98].”
Finding Global Homophily in Graph Neural Networks When Meeting Heterophily (2022, methodology): “Further, since it has been pointed out in (Liu et al., 2020) that the over-smoothing problem is caused by the coupling of neighborhood aggregation and feature transformation, we decouple these two processes by first performing feature transformation to generateH.”