Do Transformers Really Perform Badly for Graph Representation?

By Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, Tie-Yan Liu. Neural Information Processing Systems, 2021

1,491 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/acf87283fa8ae426f1a4987b345b401bf2913f61

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Hao Wang
2. Dit-Yan Yeung
3. Xingjian Shi
4. Shirui Pan
5. Jiawei Han
6. Bowen Jin
7. Qiang Yang
8. Yu Zhang

Citations in the corpus

• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (2024, methodology): “Drawing inspiration from the concept of embedding graph structures into attention as proposed in the graph neural network domain (Ying et al., 2021), we incorporate the tree structure into our attention mask, visualized in Figure 2.”
• Language is All a Graph Needs (2023, methodology): “…videos (Arnab et al., 2021) in computer vision, text in natural language processing (Singh et al., 2021), structured data in graph machine learning (Ying et al., 2021), personalized data in recommender systems (Geng et al., 2022), decision sequences in reinforcement learning (Di Palo et al.,…”
• A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection (2023, background): “Additionally, recent approaches based on graph transformer [100] or hypergraphs [121], such as [122] and [110], can capture longer-range spatial dependencies due to their global receptive field, making them a separate branch of enhanced methods.”
• Structure-Aware Transformer for Graph Representation Learning (2022, methodology): “…by encoding some carefully selected graph theoretic properties such as centrality measures and shortest path distances as positional representations (Ying et al., 2021) or by using GNNs to integrate the graph structure (Rong et al., 2020; Jain et al., 2021; Mialon et al., 2021; Shi et al., 2021).”
• Large Language Models on Graphs: A Comprehensive Survey (2023, methodology): “…typically adopt the absolute or relative positional encoding considering the position of the word token in the sequence, while graph Transformers adopt shortest path distance [141], random walk distance, the eigenvalues of the graph Laplacian [142] to consider the distance of nodes in the graph.”
• Earthformer: Exploring Space-Time Transformers for Earth System Forecasting (2022, methodology): “…natural language processing [42, 7], and later has been extended to computer vision [8, 22], multimodal text-image generation [31], graph learning [52], etc. Transformer relies on the attention mechanism to capture data correlations and is powerful at modeling complex and long-range dependencies,…”
• Long Range Graph Benchmark (2022, unknown): “Consequently, several recent works propose Graph Transformers that leverage full-connections among all nodes in the graph to capture long-range dependencies [34, 65, 44].”