Towards Deeper Graph Neural Networks with Differentiable Group Normalization

By Kaixiong Zhou, Xiao Huang, Yuening Li, Daochen Zha, Rui Chen, Xia Hu. 2020

236 citations recorded by the source corpus.

Graph neural networks (GNNs), which learn the representation of a node by aggregating its neighbors, have become an effective computational tool in downstream applications. Over-smoothing is one of the key issues which limit the performance of GNNs as the number of layers increases. It is because the stacked aggregators would make node representations converge to indistinguishable vectors. Several attempts have been made to tackle the issue by bringing linked node pairs close and unlinked pairs distinct. However, they often ignore the intrinsic community structures and would result in sub-optimal performance. The representations of nodes within the same community/class need be similar to facilitate the classification, while different classes are expected to be separated in embedding space. To bridge the gap, we introduce two over-smoothing metrics and a novel technique, i.e., differentiable group normalization (DGN). It normalizes nodes within the same group independently to increase their smoothness, and separates node distributions among different groups to significantly alleviate the over-smoothing issue. Experiments on real-world datasets demonstrate that DGN makes GNN models more robust to over-smoothing and achieves better performance with deeper GNNs.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Auto-GNN: Neural architecture search of graph neural networks (2019, methodology): “A wide variety of GNNs have been proposed to learn the node representation effectively, e.g., recursive neural networks [1, 2], graph convolutional networks [9, 30–32, 37 ] and Neural Architecture Search.”
A Survey on Oversmoothing in Graph Neural Networks (2023, unknown): “…quantify over-smoothing in deep GNNs, e.g. measures based on the Dirichlet energy on graphs (Rusch et al., 2022; Cai & Wang, 2020; Zhao & Akoglu, 2019; Zhou et al., 2021a), as well as measures based on the mean-average distance (MAD) (Chen et al., 2020a; Zhou et al., 2020), and references therein.”
3D Human Pose Estimation with Spatial and Temporal Transformers (2021, background): “Additionally, the comprehensive scaling and normalization components of the transformer are likely advantageous in mitigating the over-smoothing effect that troubles many GNN operation variants when numerous layers are stacked together [48].”
A Survey of Graph Neural Networks for Recommender Systems: Challenges, Methods, and Directions (2021, background): “Due to the over-smoothing problem, more and more studies focus on properly increasing GNN’s layers to capture higher-order connectivity correlations on graphs as well as improve models’ performance [20, 88, 150, 251].”
A Survey on Deep Semi-Supervised Learning (2021, methodology): “DGN [85] Aggregate Operation Neighborhood Normalization Hðkþ1Þ 1⁄4 HðkÞ þ Pg i1⁄41ðgið s ðkÞ i HðkÞ mi di Þ þ biÞ OðjEjÞ”
Hypergraph Contrastive Collaborative Filtering (2022, background): “The graph neural CF architecture with deeper embedding propagation layers may result in indistinguishable user vectors [25, 50], which limits the representation quality of high-order collaborative relations.”
UniGNN: a Unified Framework for Graph and Hypergraph Neural Networks (2021, background): “Furthermore, several works, like JKNet [ Xu et al. , 2018b ] , DropEdge [ Rong et al. , 2019 ] , DGN [ Zhou et al. , 2020 ] and GCNII [ Chen et al. , 2020 ] , have devoted substantial efforts to tackling the problem of over-smoothing , an issue when node embeddings in GNNs tend to converge as layers”
Benchmarking Graph Neural Networks (2023, background): “, 2020), (iv) graph-specific normalization and regularization (Chen et al., 2022; Zhou et al., 2020; Zhang et al., 2021), and (v) GNNs’ robustness and efficiency (Wei and Hu, 2022; Tailor et al.”
Training Graph Neural Networks with 1000 Layers (2021, background): “, 2020), PairNorm (Zhao & Akoglu, 2019), weight normalization (Oono & Suzuki, 2019), differentiable group normalization (Zhou et al., 2020), and GraphNorm (Cai et al.”