Rethinking Graph Transformers with Spectral Attention

By Devin Kreuzer, Dominique Beaini, William L. Hamilton, Vincent L'etourneau, Prudencio Tossou. Neural Information Processing Systems, 2021

808 citations recorded by the source corpus.

In recent years, the Transformer architecture has proven to be very successful in sequence processing, but its application to other data structures, such as graphs, has remained limited due to the difficulty of properly defining positions. Here, we present the $\textit{Spectral Attention Network}$ (SAN), which uses a learned positional encoding (LPE) that can take advantage of the full Laplacian spectrum to learn the position of each node in a given graph. This LPE is then added to the node features of the graph and passed to a fully-connected Transformer. By leveraging the full spectrum of the Laplacian, our model is theoretically powerful in distinguishing graphs, and can better detect similar sub-structures from their resonance. Further, by fully connecting the graph, the Transformer does not suffer from over-squashing, an information bottleneck of most GNNs, and enables better modeling of physical phenomenons such as heat transfer and electric interaction. When tested empirically on a set of 4 standard datasets, our model performs on par or better than state-of-the-art GNNs, and outperforms any attention-based model by a wide margin, becoming the first fully-connected architecture to perform well on graph benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Materials Chemistry

View Paper: https://www.semanticscholar.org/paper/5863d7b35ea317c19f707376978ef1cc53e3534c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Materials Chemistry

Citations in the corpus

Structure-Aware Transformer for Graph Representation Learning (2022, methodology): “SAN (Kreuzer et al., 2021) also used the Laplacian eigenvectors for computing an absolute encoding, but computed attention on the full graph, while distinguishing between true and created edges.”
Graph Neural Networks with Learnable Structural and Positional Representations (2021, methodology): “During the training, eigenvectors are uniformly sampled at random between the 2 possibilities (Dwivedi et al., 2020; Kreuzer et al., 2021) in order for the network to learn to be invariant w.”
Recipe for a General, Powerful, Scalable Graph Transformer (2022, methodology): “We compare GPS against a set of popular message-passing neural networks (GCN, GIN, GatedGCN, PNA, etc.), graph transformers (SAN, Graphormer, etc.), and other recent graph neural networks with SOTA results (CIN, CRaWL, GIN-AK+, ExpC).”
Benchmarking Graph Neural Networks (2023, methodology): “This insight has been received keenly in the recent literature and there are a number of works that propose positional encoding schemes with some addressing the challenges of using Laplacian eigenvectors (Kreuzer et al., 2021; Wang et al., 2022; Lim et al., 2022).”
Do Transformers Really Perform Bad for Graph Representation? (2021, methodology): “Specially, except Graphormer, [54] 509,549 0.526 ± 0.051 GraphSage [18] 505,341 0.398 ± 0.002 GAT [50] 531,345 0.384 ± 0.007 GCN [26] 505,079 0.367 ± 0.011 GatedGCN-PE [4] 505,011 0.214 ± 0.006 MPNN (sum) [15] 480,805 0.145 ± 0.007 PNA [10] 387,155 0.142 ± 0.010 GT [13] 588,929 0.226 ± 0.014 SAN [28”
Long Range Graph Benchmark (2022, neutral): “By design, fully connected Graph Transformers [65, 34, 44] are able to model long-range dependencies in the graphs and alleviate information bottleneck to some extent [53].”
EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations (2023, neutral): “SphereNet [48] and GemNet [50, 51] further include dihedral angles by considering quadruplet representations.”
Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs (2022, background): “A parallel line of research focuses on applying Transformer networks [78] to other domains like computer vision [9, 16, 73] and graph [18, 44, 85, 66] and has demonstrated widespread success.”