Graph-Bert: Only Attention is Needed for Learning Graph Representations

By Jiawei Zhang, Haopeng Zhang, Li Sun, Congying Xia. arXiv.org, 2020

391 citations recorded by the source corpus.

The dominant graph neural networks (GNNs) over-rely on the graph links, several serious performance problems with which have been witnessed already, e.g., suspended animation problem and over-smoothing problem. What's more, the inherently inter-connected nature precludes parallelization within the graph, which becomes critical for large-sized graph, as memory constraints limit batching across the nodes. In this paper, we will introduce a new graph neural network, namely GRAPH-BERT (Graph based BERT), solely based on the attention mechanism without any graph convolution or aggregation operators. Instead of feeding GRAPH-BERT with the complete large input graph, we propose to train GRAPH-BERT with sampled linkless subgraphs within their local contexts. GRAPH-BERT can be learned effectively in a standalone mode. Meanwhile, a pre-trained GRAPH-BERT can also be transferred to other application tasks directly or with necessary fine-tuning if any supervised label information or certain application oriented objective is available. We have tested the effectiveness of GRAPH-BERT on several graph benchmark datasets. Based the pre-trained GRAPH-BERT with the node attribute reconstruction and structure recovery tasks, we further fine-tune GRAPH-BERT on node classification and graph clustering tasks specifically. The experimental results have demonstrated that GRAPH-BERT can out-perform the existing GNNs in both the learning effectiveness and efficiency.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/78542c2be9bb853a4e04642f2d315cfb0c6d94b3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Self-Supervised Learning on Graphs: Contrastive, Generative, or Predictive (2021, background): “cess [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], it is still very challenging to deal with the inherent differences between grid-like and structured-like data.”
A Generalization of Transformer Networks to Graphs (2020, methodology): “The WL-PE which are absolute structural roles of nodes in the original graph computed using WL algorithm (Zhang et al. 2020; Niepert, Ahmed, and Kutzkov 2016), are not variant to the subgraphs and can be easily used as a generic PE mechanism.”
Text Graph Transformer for Document Classification (2020, methodology): “Weisfeiler-Lehman Structural Encoding We adopt the WL Role Embedding by (Zhang et al., 2020a) to capture the structure of text graph.”
Representing Long-Range Context for Graph Neural Networks with Global Attention (2022, methodology): “Recent works such as Zhang et al. [38], Rong et al. [24], and Dwivedi and Bresson [12] propose GNN layers that let nodes attend to other nodes in some surrounding neighborhood via Transformer-style attention, whereas we use self attention for a permutation-invariant, graph-level pooling or…”
Graph Contrastive Learning with Augmentations (2020, background): “If extended to pre-training GCNs, this framework can potentially overcome the aforementioned limitations of proximity-based pre-training methods [22, 23, 34, 35, 36, 37, 38, 39].”
Structure-Aware Transformer for Graph Representation Learning (2022, methodology): “…methods include the Laplacian positional encoding (Dwivedi & Bresson, 2021; Kreuzer et al., 2021), Weisfeiler–Lehman-based positional encoding (Zhang et al., 2020), and random walk positional encoding (RWPE) (Li et al., 2020b; Dwivedi et al., 2022), while distance-based methods include…”
Do Transformers Really Perform Bad for Graph Representation? (2021, methodology): “There are various PEs employed by previous Transformer-based GNNs, e.g., Weisfeiler-Lehman-PE (WL-PE) [61] and Laplacian PE [3, 14].”
Do Transformers Really Perform Badly for Graph Representation? (2021, methodology): “, Weisfeiler-LehmanPE (WL-PE) [55] and Laplacian PE [3, 14].”