Representing Long-Range Context for Graph Neural Networks with Global Attention

By Zhanghao Wu, Paras Jain, Matthew A. Wright, Azalia Mirhoseini, Joseph E. Gonzalez, Ion Stoica. Neural Information Processing Systems, 2022

423 citations recorded by the source corpus.

Graph neural networks are powerful architectures for structured datasets. However, current methods struggle to represent long-range dependencies. Scaling the depth or width of GNNs is insufficient to broaden receptive fields as larger GNNs encounter optimization instabilities such as vanishing gradients and representation oversmoothing, while pooling-based approaches have yet to become as universally useful as in computer vision. In this work, we propose the use of Transformer-based self-attention to learn long-range pairwise relationships, with a novel"readout"mechanism to obtain a global graph embedding. Inspired by recent computer vision results that find position-invariant attention performant in learning long-range relationships, our method, which we call GraphTrans, applies a permutation-invariant Transformer module after a standard GNN module. This simple architecture leads to state-of-the-art results on several graph classification tasks, outperforming methods that explicitly encode graph structure. Our results suggest that purely-learning-based approaches without graph structure may be suitable for learning high-level, long-range relationships on graphs. Code for GraphTrans is available at https://github.com/ucbrise/graphtrans.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Materials Chemistry

View Paper: https://www.semanticscholar.org/paper/2b8a207189bc02d73d1dce850bcde24dbd984483

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Materials Chemistry

Citations in the corpus

Structure-Aware Transformer for Graph Representation Learning (2022, methodology): “, 2021) or by using GNNs to integrate the graph structure (Rong et al., 2020; Jain et al., 2021; Mialon et al., 2021; Shi et al., 2021).”
Recipe for a General, Powerful, Scalable Graph Transformer (2022, methodology): “Recent 2-step methods GraphTrans [29] and SAT [8] can use edge features in their first MPNN step, however this step is applied only once and typically includes several k rounds of message passing.”