Heterogeneous Graph Transformer

By Ziniu Hu, Yuxiao Dong, Kuansan Wang, Yizhou Sun. The Web Conference, 2020

1,684 citations recorded by the source corpus.

Recent years have witnessed the emerging success of graph neural networks (GNNs) for modeling structured data. However, most GNNs are designed for homogeneous graphs, in which all nodes and edges belong to the same types, making it infeasible to represent heterogeneous structures. In this paper, we present the Heterogeneous Graph Transformer (HGT) architecture for modeling Web-scale heterogeneous graphs. To model heterogeneity, we design node- and edge-type dependent parameters to characterize the heterogeneous attention over each edge, empowering HGT to maintain dedicated representations for different types of nodes and edges. To handle Web-scale graph data, we design the heterogeneous mini-batch graph sampling algorithm—HGSampling—for efficient and scalable training. Extensive experiments on the Open Academic Graph of 179 million nodes and 2 billion edges show that the proposed HGT model consistently outperforms all the state-of-the-art GNN baselines by 9–21 on various downstream tasks. The dataset and source code of HGT are publicly available at https://github.com/acbull/pyHGT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems

View Paper: https://www.semanticscholar.org/paper/0ca7d8c3250d43d14fdde46bf6fc299654d861ef

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

Citations in the corpus

Heterogeneous Graph Structure Learning for Graph Neural Networks (2021, background): “…follow a message-passing scheme where the node embedding is learned by aggregating and transforming the embeddings of its original neighbors (Zhang et al. 2019a; Hu et al. 2020; Zhao et al. 2020a; Hong et al. 2020) or metapath-based neighbors (Wang et al. 2019b; Yun et al. 2019; Fu et al. 2020).”
Knowledge Graph Prompting for Multi-Document Question Answering (2023, background): “Following (Hu et al. 2020), let G = (V, E) be a knowledge graph constructed from a set of documents D, where the node set V = {vi}i=1 representing document structures (e.”
GPT-GNN: Generative Pre-Training of Graph Neural Networks (2020, methodology): “On the OAG and Amazon datasets, we use the state-of-the-art heterogeneous GNN—Heterogeneous Graph Transformer (HGT) [15]—as the base model for GPT-GNN.”
Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning (2021, background): “HGT [15] is designed for Web-scale heterogeneous networks.”
Do Transformers Really Perform Bad for Graph Representation? (2021, background): “In this section, we highlight the most recent works which attempt to develop standard Transformer architecture-based GNN or graph structural encoding, but spend less effort on elaborating the works by adapting attention mechanism to GNNs [32, 55, 7, 22, 1, 47, 48, 56, 45].”
CGLB: Benchmark Tasks for Continual Graph Learning (2022, background): “…including Heterogeneous Graph Neural Network (HetGNN) [62], Heterogeneous Graph Attention Network (HAN) [49], Heterogeneous Graph Transformer (HGT) [22], Relational Graph Convolutional Networks (R-GCNs) [41], Metapath2vec [11], and Predictive Text Embedding (PTE) [44] will be implemented into our…”
Graph Representation Learning in Biomedicine and Healthcare (2021, positive): “Recent architectures have leveraged dynamic message passing 72,85,86 to deal with evolving and time-varying graphs, as well as few-shot learning 87 or self-supervised strategies 88,89 to deal with graphs that are poorly annotated and that have limited information about labels.”
Do Transformers Really Perform Badly for Graph Representation? (2021, background): “, feature aggregation) in classic GNN variants by the softmax attention [46, 7, 23, 47, 55, 42, 13].”
DiffMG: Differentiable Meta Graph Search for Heterogeneous Graph Neural Networks (2020, methodology): “, GTN [32] and HGT [12], implicitly learn meta paths by fusing different types based on attention mechanisms.”
Are we really making much progress?: Revisiting, benchmarking and refining heterogeneous graph neural networks (2021, methodology): “Take node classification for example, numerous HGNNs, such as HAN [36], GTN [43], RSHN [45], HetGNN [44], MAGNN [12], HGT [20], and HetSANN [17] were developed within the last two years.”
Self-Supervised Learning: Generative or Contrastive (2020, neutral): “) and graph learning (node classification [53], [64], [95], [126], graph classification [7], [110], [146] etc.”
GraphGPT: Graph Instruction Tuning for Large Language Models (2023, neutral): “Graphtransformernetworks(GTNs)[15,61]incor-porateself-attentionandpositionalencodingtocaptureglobal dependenciesandstructuralpatternsinthegraph.”
Text Graph Transformer for Document Classification (2020, methodology): “Recently, Transformer (Vaswani et al., 2017) models have been applied in novel GNN designs (Hu et al., 2020; Zhang et al., 2020a).”
OAG-BERT: Pre-train Heterogeneous Entity-augmented Academic Language Models (2021, methodology): “Specifically, we take the heterogeneous graph transformer (HGT) model from [17] and combine it with the pre-trained embeddings from OAG-BERT.”
Extracting Temporal Event Relation with Syntactic-Guided Temporal Graph Transformer (2021, methodology): “Our work is also related to the variants of Graph Neural Networks (GNN) (Kipf and Welling, 2016; Veliˇckovi´c et al., 2017; Zhou et al., 2018), especially Graph Transformers (Yun et al., 2019; Chen et al., 2019; Hu et al., 2020; Wang et al., 2020b).”
Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research (2022, result): “Heterogeneity in transaction graphs975 was captured whereas the presented methodology outper-976 formed previousmodels HGT [191] andGEM [192].”
A Survey on Heterogeneous Graph Embedding: Methods, Techniques, Applications and Sources (2020, neutral): “Then a series of attention-based HGNNs were proposed [78], [79], [80], [81].”
V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer (2022, background): “Inspired by HGT, we build a customized heterogeneous multi-head self-attention module tailored for graph attribute-aware multi-agent 3D visual feature fusion, which is able to capture the heterogeneity of V2X systems.”
A Generalization of Transformer Networks to Graphs (2020, background): “There is another trans-former based approach developed for heterogeneous information networks, namely Heterogeneous Graph Trans-former (HGT) by Hu et al. (2020).”