Do Transformers Really Perform Bad for Graph Representation?

By Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, Tie-Yan Liu. arXiv.org, 2021

597 citations recorded by the source corpus.

The Transformer architecture has become a dominant choice in many domains, such as natural language processing and computer vision. Yet, it has not achieved competitive performance on popular leaderboards of graph-level prediction compared to mainstream GNN variants. Therefore, it remains a mystery how Transformers could perform well for graph representation learning. In this paper, we solve this mystery by presenting Graphormer, which is built upon the standard Transformer architecture, and could attain excellent results on a broad range of graph representation learning tasks, especially on the recent OGB Large-Scale Challenge. Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model. To this end, we propose several simple yet effective structural encoding methods to help Graphormer better model graph-structured data. Besides, we mathematically characterize the expressive power of Graphormer and exhibit that with our ways of encoding the structural information of graphs, many popular GNN variants could be covered as the special cases of Graphormer.

Subfields: Artificial Intelligence, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/47ae807cd511b35e78a2cd4e198283dea6dafd41

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

Dense-to-Sparse Gate for Mixture-of-Experts (2021, methodology): “Transformer models have achieved a huge success in improving model quality via scaling data and model sizes (Brown et al., 2020; Dosovitskiy et al., 2021; Ying et al., 2021).”
AlphaFold2-aware protein-DNA binding site prediction using graph transformer (2022, background): “In the last few years, transformer variants have also been shown great performance in graph representation learning [40-42].”
Graph Neural Networks with Learnable Structural and Positional Representations (2021, methodology): “However, these methods either use non-learnable PEs to encode graph structure information (Dwivedi & Bresson, 2021; Ying et al., 2021; Mialon et al., 2021), or inject learned PEs to the Transformer network that relies on Laplacian eigenvectors (Kreuzer et al., 2021), thus inheriting the sign…”
An end-to-end deep learning method for rotamer-free protein side-chain packing (2022, methodology): “We choose to incorporate this information at the input level, rather than in each attention block (as is proposed in [35]).”
OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs (2021, background): “For a more direct comparison, with a single model, the Machine Learning reports the validation MAE of 0.097 for their Graphormer model (Ying et al., 2021a), which is 0.04 points lower than our best GIN-VIRTUAL baseline.”
Uni-Mol: A Universal 3D Molecular Representation Learning Framework (2023, methodology): “In most current MRL methods, one starts with representing molecules as 1D sequential strings, such as SMILES [16, 17, 18] and InChI [19, 20, 21], or 2D graphs [22, 11, 23, 12, 24].”
Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs (2022, methodology): “With irreps features, we build upon existing generic GNNs and Transformer networks [78, 16, 85, 45, 47] by incorporating equivariant operations like tensor products.”
Recipe for a General, Powerful, Scalable Graph Transformer (2022, methodology): “There have been many recent works on PE and SE, notably on Laplacian PE [13, 34, 3, 36], shortest-path and node degree centrality [59], kernel distance [41], random-walk SE [15], structure-aware [8, 6, 5], and more.”
Benchmarking Graph Neural Networks (2023, methodology): “After the introduction of Laplacian PE through this benchmark, new ideas followed up in the literature for improving PE (Beaini et al., 2021; Wang et al., 2022; Lim et al., 2022; Kreuzer et al., 2021; Ying et al., 2021; Mialon et al., 2021), thus demonstrating how the identification of first princip”
MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design (2022, methodology): “To adapt it to molecular graphs, recent works utilize the attention to better capture the correlation at node, subgraph, and graph-level, including GAT [40], AttentiveFP [41], GTransformer [42], AWARE [43], and Graphormer [44].”