Understanding Attention and Generalization in Graph Neural Networks

By Boris Knyazev, Graham W. Taylor, Mohamed R. Amer. Neural Information Processing Systems, 2019

403 citations recorded by the source corpus.

We aim to better understand attention over nodes in graph neural networks (GNNs) and identify factors influencing its effectiveness. We particularly focus on the ability of attention GNNs to generalize to larger, more complex or noisy graphs. Motivated by insights from the work on Graph Isomorphism Networks, we design simple graph reasoning tasks that allow us to study attention in a controlled environment. We find that under typical conditions the effect of attention is negligible or even harmful, but under certain conditions it provides an exceptional gain in performance of more than 60% in some of our classification tasks. Satisfying these conditions in practice is challenging and often requires optimal initialization or supervised training of attention. We propose an alternative recipe and train attention in a weakly-supervised fashion that approaches the performance of supervised models, and, compared to unsupervised models, improves results on several synthetic as well as real datasets. Source code and datasets are available at this https URL.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/01cf1d83b7a4983001a50ec274043f53783ee94e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Interpretable and Generalizable Graph Learning via Stochastic Attention Mechanism (2022, methodology): “For MNIST-75sp, we use the default splits given by (Knyazev et al., 2019); due to its large size in the graph setting, we also reduce the number of training samples following (Wu et al.”
Graph-Refined Convolutional Network for Multimedia Recommendation with Implicit Feedback (2020, result): “suggest is that GAT heavily affected by the initialization of ID embeddings, which is consistent with the finding [21].”
TUDataset: A collection of benchmark datasets for learning with graphs (2020, methodology): “Knyazev et al. (2019) introduced the datasets C OLORS and T RIANGLES , where the task is to count the number of nodes with a given one-hot-encoded color and the number of triangles, respectively.”
How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision (2022, background): “Although conventional attention is trained without direct supervision, if we have prior knowledge about what to attend, we can supervise attention using them (Knyazev et al., 2019; Yu et al., 2017).”
Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks (2020, background): “It is perhaps surprising that the attention mechanism of GAT improved very little on the GCN result despite the large neighborhoods—one reason might be that it is difﬁcult to learn what to attend to in the semi-supervised setting, as discussed in-depth by Knyazev et al. (2019).”
Discovering Invariant Rationales for Graph Neural Networks (2022, methodology): “Among them, graph attention (Veliˇckovi´c et al., 2018) and pooling (Lee et al., 2019; Knyazev et al., 2019; Gao & Ji, 2019; Ranjan et al., 2020) operators prevail, which work as a computational block of a GNN to generate soft or hard masks on the input graph.”
Benchmarking Graph Neural Networks (2023, neutral): “We use SLIC super-pixels from (Knyazev et al., 2019)10.”
Fine-grained Fact Verification with Kernel Graph Attention Network (2019, background): “The readout module (Knyazev et al., 2019) calculates the probability P (np|G) and attentively combines per-node signals for prediction.”