A Note on Over-Smoothing for Graph Neural Networks

By Chen Cai, Yusu Wang. arXiv.org, 2020

374 citations recorded by the source corpus.

Graph Neural Networks (GNNs) have achieved a lot of success on graph-structured data. However, it is observed that the performance of graph neural networks does not improve as the number of layers increases. This effect, known as over-smoothing, has been analyzed mostly in linear cases. In this paper, we build upon previous results \cite{oono2019graph} to further analyze the over-smoothing effect in the general graph neural network architecture. We show when the weight matrix satisfies the conditions determined by the spectrum of augmented normalized Laplacian, the Dirichlet energy of embeddings will converge to zero, resulting in the loss of discriminative power. Using Dirichlet energy to measure "expressiveness" of embedding is conceptually clean; it leads to simpler proofs than \cite{oono2019graph} and can handle more non-linearities.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Materials Chemistry

View Paper: https://www.semanticscholar.org/paper/c6d550c3fcecf27b979be84c4cd444cc1c72bf47

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Language is All a Graph Needs (2023, methodology): “These methods often require training multiple models, incurring significant computational overhead and tend to easily inherit drawbacks of GNNs such as over-smoothing (Cai and Wang, 2020).”
A Survey on Oversmoothing in Graph Neural Networks (2023, neutral): “…has focused on precisely deﬁning over-smoothing through measures of node feature similarities such as the the graph Dirichlet (Rusch et al., 2022; Cai & Wang, 2020; Zhao & Akoglu, 2019; Zhou et al., 2021a), cosine similarity (Chen et al., 2020a), and other related similarity scores (Zhou et al.,…”
Intrinsic-Extrinsic Convolution and Pooling for Learning on 3D Protein Structures (2021, background): “However, GCNN approaches suffer from over-smoothing, i. e., indistinguishable node representations after stacking several layers, which limits the maximum depth usable for such architectures (Cai & Wang, 2020).”
Length-Induced Embedding Collapse in PLM-based Models (2024, unknown): “Viewing the attention matrix as an adjacency matrix, a higher temperature leads to a denser graph, enhancing information exchange between nodes but losing high-frequency details (Oono and Suzuki, 2020; Cai and Wang, 2020).”