Graph Transformer Networks

By Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, Hyunwoo J. Kim. Neural Information Processing Systems, 2019

1,382 citations recorded by the source corpus.

Graph neural networks (GNNs) have been widely used in representation learning on graphs and achieved state-of-the-art performance in tasks such as node classification and link prediction. However, most existing GNNs are designed to learn node representations on the fixed and homogeneous graphs. The limitations especially become problematic when learning representations on a misspecified graph or a heterogeneous graph that consists of various types of nodes and edges. In this paper, we propose Graph Transformer Networks (GTNs) that are capable of generating new graph structures, which involve identifying useful connections between unconnected nodes on the original graph, while learning effective node representation on the new graphs in an end-to-end fashion. Graph Transformer layer, a core layer of GTNs, learns a soft selection of edge types and composite relations for generating useful multi-hop connections so-call meta-paths. Our experiments show that GTNs learn new graph structures, based on data and tasks without domain knowledge, and yield powerful node representation via convolution on the new graphs. Without domain-specific graph preprocessing, GTNs achieved the best performance in all three benchmark node classification tasks against the state-of-the-art methods that require pre-defined meta-paths from domain knowledge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/aa63ac11aa9dcaa9edd4c88db18bec87e0834328

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Deep Generative Models in Engineering Design: A Review (2021, methodology): “Graph neural networks (GNNs) [42,43,44,45,46] provide an excellent tool for machine learning on graphs.”
Video Graph Transformer for Video Question Answering (2022, background): “, node degrees) into self-attention block of Transformer [66], or designing transformer -style convolution blocks to fuse information from heterogeneous graphs [69].”
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks (2023, background): “Notably, transformers have showcased superior capabilities compared to Graph Neural Networks (GNNs) in constructing meta-paths from different types of edges in heterogeneous graphs (Yun et al., 2019).”
Flexible Job-Shop Scheduling via Graph Neural Network and Deep Reinforcement Learning (2023, background): “While some HGNNs have been proposed recently [36]–[38], they do not consider the unique properties of the FJSP heterogeneous graph H t .”
A Generalization of Transformer Networks to Graphs (2020, background): “Yun et al. (2019) developed Graph Transformer Networks (GTN) to learn on heterogeneous graphs with a target to transform a given heterogeneous graph into a meta-path based graph and then perform convolution.”
AWB-GCN: A Graph Convolutional Network Accelerator with Runtime Workload Rebalancing (2019, background): “And although AWB-GCN is designed for GCNs, it is generally efﬁcient for GNNs whose major arithmetic primitives are also SpMMs, e.g., GraphSage [66], GINConv [67], and GTN [30].”
Generalization and Representational Limits of Graph Neural Networks (2020, background): “, 2005), in their various incarnations, have emerged as models of choice for embedding graph-structured data from a diverse set of domains, including molecular structures, knowledge graphs, biological networks, social networks, and n-body problems (Duvenaud et al., 2015; Defferrard et al., 2016; Bat”
Extracting Temporal Event Relation with Syntactic-Guided Temporal Graph Transformer (2021, methodology): “Our work is also related to the variants of Graph Neural Networks (GNN) (Kipf and Welling, 2016; Veliˇckovi´c et al., 2017; Zhou et al., 2018), especially Graph Transformers (Yun et al., 2019; Chen et al., 2019; Hu et al., 2020; Wang et al., 2020b).”
A Survey of Knowledge-enhanced Text Generation (2020, background): “slation KGENMT [177] DBpedia [3] TransE [13] Question answering MHPGM [7] ConceptNet [133] PR [76, 77] Creative writing StoryEnding [56] ConceptNet [133] GAT [140] GraphWriter [75] An abstract KG GTN [172] PaperRobot [147] A biology KG TransE [13] GAT [140] objective, to preserve inherent structures”
A Survey on Heterogeneous Graph Embedding: Methods, Techniques, Applications and Sources (2020, neutral): “GTN [83] designs an aggregation function, which can ﬁnd the suitable meta-paths automatically during the process of message passing.”
Transformers learn in-context by gradient descent (2022, methodology): “, 2017) have demonstrated their superiority in numerous benchmarks and various fields of modern machine learning, and have emerged as the de-facto neural network architecture used for modern AI (Dosovitskiy et al., 2021; Yun et al., 2019; Carion et al., 2020; Gulati et al., 2020).”
Do Transformers Really Perform Bad for Graph Representation? (2021, background): “In this section, we highlight the most recent works which attempt to develop standard Transformer architecture-based GNN or graph structural encoding, but spend less effort on elaborating the works by adapting attention mechanism to GNNs [32, 55, 7, 22, 1, 47, 48, 56, 45].”
Heterogeneous Graph Structure Learning for Graph Neural Networks (2021, methodology): “…, denoted as S Featr 2 j V (cid:30)h ( r ) j(cid:2)j V (cid:30)t ( r ) j , can be obtained by fusing these graphs through a channel attention layer (Yun et al. 2019): where is the stacked matrix of the feature candidate graphs, and (cid:9) Featr denotes a channel attention layer with parameters W…”
Heterogeneous Graph Transformer (2020, background): “ctures, such as PathSim [18] and metapath2vec [3]. Recently, in view of graph neural networks’ (GNNs) success [7, 9, 22], there are several attempts to adopt GNNs to learn with heterogeneous networks [14, 23, 26, 27]. However, these works face several issues: First, most of them involve the design o”
Propagation Model Search for Graph Neural Networks (2020, methodology): “Interestingly, these paths are the top three ranked by attention in the GTN paper (see Table 3 in (Yun et al. 2019)).”