How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision

By Dongkwan Kim, Alice H. Oh. International Conference on Learning Representations, 2022

316 citations recorded by the source corpus.

Attention mechanism in graph neural networks is designed to assign larger weights to important neighbor nodes for better representation. However, what graph attention learns is not understood well, particularly when graphs are noisy. In this paper, we propose a self-supervised graph attention network (SuperGAT), an improved graph attention model for noisy graphs. Specifically, we exploit two attention forms compatible with a self-supervised task to predict edges, whose presence and absence contain the inherent information about the importance of the relationships between nodes. By encoding edges, SuperGAT learns more expressive attention in distinguishing mislinked neighbors. We find two graph characteristics influence the effectiveness of attention forms and self-supervision: homophily and average degree. Thus, our recipe provides guidance on which attention design to use when those two graph characteristics are known. Our experiment on 17 real-world datasets demonstrates that our recipe generalizes across 15 datasets of them, and our models designed by recipe show improved performance over baselines.

Subfields: Artificial Intelligence, Materials Chemistry

View Paper: https://www.semanticscholar.org/paper/dc9b9c5a76df331409890eea0d01e3cf3459a526

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

Citations in the corpus

Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods (2021, background): “Various GNNs have been proposed to achieve higher performance in low-homophily settings [82, 44, 81, 17, 15, 73, 36, 35].”
Graph Self-Supervised Learning: A Survey (2021, background): “SuperGAT [49] further extends this idea to every layers in the encoder.”
How Attentive are Graph Attention Networks? (2021, methodology): “This is consistent with Kim and Oh (2021), who found that DPGAT predicts edge presence better, but in much smaller datasets (Cora, CiteSeer, PubMed, and PPI).”
Language is All a Graph Needs (2023, methodology): “We consider top-ranked GNNs from the corresponding leaderboard 4 , including SAIL, ALT-OPT, GRAND, etc., as the GNN baselines (Luan et al., 2019; Kim and Oh, 2022; Feng et al., 2020; Han et al., 2023a; Yu et al., 2022b).”
Large-Scale Representation Learning on Graphs via Bootstrapping (2021, unknown): “This aligns with recent observations (Kim & Oh, 2021; Wang et al., 2019) that auxiliary losses must be chosen carefully for the stability of GAT attention weights.”
Bootstrapped Representation Learning on Graphs (2021, background): “This aligns with recent observations that carefully chosen auxiliary losses are often paramount for stability of GAT models (Kim & Oh, 2021; Wang et al., 2019).”
Self-Supervised Learning of Graph Neural Networks: A Unified Review (2021, unknown): “In addition, a recent work SuperGAT [75] includes the GAE objective as a selfsupervised auxiliary loss during training a graph attention network to guide the learning of more expressive attention operators.”