PairNorm: Tackling Oversmoothing in GNNs

By Lingxiao Zhao, Leman Akoglu. 2019

611 citations recorded by the source corpus.

The performance of graph neural nets (GNNs) is known to gradually decrease with increasing number of layers. This decay is partly attributed to oversmoothing, where repeated graph convolutions eventually make node embeddings indistinguishable. We take a closer look at two different interpretations, aiming to quantify oversmoothing. Our main contribution is PairNorm, a novel normalization layer that is based on a careful analysis of the graph convolution operator, which prevents all node embeddings from becoming too similar. What is more, PairNorm is fast, easy to implement without any change to network architecture nor any additional parameters, and is broadly applicable to any GNN. Experiments on real-world graphs demonstrate that PairNorm makes deeper GCN, GAT, and SGC models more robust against oversmoothing, and significantly boosts performance for a new problem setting that benefits from deeper GNNs. Code is available at this https URL.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jianfeng Gao
3. Xia Hu
4. Jie Tang
5. Michael M. Bronstein
6. Xinchao Wang
7. Mohit Bansal
8. Percy Liang

In Statistical and Nonlinear Physics

1. Xia Hu
2. Wenwu Zhu
3. Xiao Huang
4. Jie Tang
5. Michael M. Bronstein
6. Yuxiao Dong
7. Qingquan Song
8. Xinchao Wang

Citations in the corpus

• Auto-GNN: Neural architecture search of graph neural networks (2019, methodology): “For deep GNNs, we consider state-of-the-art (SOTA) models of PairNorm (Zhao and Akoglu, 2019), SGC (Wu et al.”
• Implicit Graph Neural Networks (2020, background): “, 2020) and theoretical (Oono and Suzuki, 2020; Zhao and Akoglu, 2020) works follows on the over-smoothing phenomenon.”
• Principal Neighbourhood Aggregation for Graph Nets (2020, methodology): “STD is not used in the GNN literature.”
• Graph Random Neural Networks for Semi-Supervised Learning on Graphs (2020, background): “However, most existing GNNs inherently suffer from the limitations of over-smoothing [6, 23, 24, 30], non-robustness [48, 45], and weak-generalization when labeled nodes are scarce.”
• How Attentive are Graph Attention Networks? (2021, methodology): “Self-supervision (Kim and Oh, 2021; Rong et al., 2020a; Hassani and Khasahmadi, 2020), graph regularization methods (Wang et al., 2019a; Zhao and Akoglu, 2020; Rong et al., 2020b), and other tricks (Wang, 2021; Huang et al., 2021) are orthogonal to the contribution of the GNN layer itself, and…”
• Learning to Drop: Robust Graph Neural Network via Topological Denoising (2020, background): “DropEdge and PairNorm investigated the over-smoothing problem of stacking multiple GNN layers [38, 55].”
• A Survey on Oversmoothing in Graph Neural Networks (2023, positive): “Another example of implicit regularization is PairNorm (Zhao & Akoglu, 2019), where the pairwise distances are set to be constant throughout every layer in the deep GNN.”
• Finding Global Homophily in Graph Neural Networks When Meeting Heterophily (2022, background): “Further, GNNs have been widely studied from various perspectives, such as the over-smoothing problem (Zhao & Akoglu, 2019; Rong et al., 2019), the adversarial attack and defense (Dai et al., 2018; Zhu et al., 2019), and the model explainability (Vu & Thai, 2020; Shan et al., 2021).”
• Amalgamating Knowledge from Heterogeneous Graph Neural Networks (2021, background): “Furthermore, [52] and [29] propose a novel PairNorm layer and a DropEdge strategy to alleviate the oversmoothing problem in GNNs.”
• DeeperGCN: All You Need to Train Deeper GCNs (2020, background): “Recent works focus on addressing this phenomenon [Klicpera et al., 2019, Rong et al., 2020, Zhao and Akoglu, 2020].”
• HittER: Hierarchical Transformers for Knowledge Graph Embeddings (2020, background): “tions (Zhao and Akoglu, 2020; Zhou et al., 2020) and efficiency issues when scaling up (Rossi et al.”
• Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction (2021, background): “Although it has been reported that GNNs with deep layers often result in significant performance drop [36, 77], we observe NBFNet does not have this issue.”
• Graph Neural Networks Exponentially Lose Expressive Power for Node Classification (2019, background): “Several studies gave theoretical explanations of the over-smoothing phenomena for linear GNNs (Li et al., 2018; Zhang, 2019; Zhao & Akoglu, 2020).”
• Towards Deeper Graph Neural Networks with Differentiable Group Normalization (2020, methodology): “Measurements in existing work are mainly based on the distances between node pairs [20, 24].”
• A Note on Over-Smoothing for Graph Neural Networks (2020, methodology): “How to extend our strategy to more general graph learning such as other nonlinearities, normalization strategy (Zhao & Akoglu, 2019), graphs with both node and edge features and attention mechanism (Veliˇckovi´c et al., 2017)”
• On the Bottleneck of Graph Neural Networks and its Practical Implications (2020, positive): “Several approaches were proposed to mitigate over-smoothing, such as normalization (Zhao and Akoglu, 2020), edge-based dropout (Rong et al., 2020), and noise-reduction regularization (Chen et al., 2020).”
• SIGN: Scalable Inception Graph Neural Networks (2020, background): “Significant effort has recently been devoted to methods allowing to design deep graph neural networks with many graph convolutional layers [62, 20, 38, 68, 50], which otherwise appear difficult to train [39, 33, 60].”
• Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks (2021, methodology): “There are various empirical solutions for the oversmoothing problem [5] of GCNs: residual connections and dilated convolutions [25]; new normalization strategies [21] and aggregations [7]; and edge dropout [26].”