Graph Attention Networks
By Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio', Yoshua Bengio. International Conference on Learning Representations, 2017
27,282 citations recorded by the source corpus.
We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over their neighborhoods' features, we enable (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of costly matrix operation (such as inversion) or depending on knowing the graph structure upfront. In this way, we address several key challenges of spectral-based graph neural networks simultaneously, and make our model readily applicable to inductive as well as transductive problems. Our GAT models have achieved or matched state-of-the-art results across four established transductive and inductive graph benchmarks: the Cora, Citeseer and Pubmed citation network datasets, as well as a protein-protein interaction dataset (wherein test graphs remain unseen during training).
Subfields: Artificial Intelligence, Materials Chemistry, Molecular Biology, Model Architectures, Graph Learning
View Paper: https://arxiv.org/abs/1710.10903
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Percy Liang
- Quoc V. Le
- Sergey Levine
- Luke S. Zettlemoyer
- Ruslan Salakhutdinov
In Materials Chemistry
- William L. Hamilton
- Tommi S. Jaakkola
- Regina Barzilay
- Jure Leskovec
- Jian Tang
- Marinka Zitnik
- Wengong Jin
- Yoshua Bengio
In Molecular Biology
- William L. Hamilton
- Jure Leskovec
- Tommi S. Jaakkola
- Stephan Günnemann
- Rex Ying
- Regina Barzilay
- Marinka Zitnik
- Aleksandar Bojchevski
In Model Architectures
- Yoshua Bengio
- Quoc V. Le
- Ruslan Salakhutdinov
- Mohit Bansal
- Kyunghyun Cho
- Jianfeng Gao
- J. Urgen Schmidhuber
- Aaron C. Courville
In Graph Learning
- William L. Hamilton
- Jure Leskovec
- Stephan Günnemann
- Matthias Fey
- Michael M. Bronstein
- Rex Ying
- Aleksandar Bojchevski
- Marinka Zitnik
Citations in the corpus
- Rethinking Attention with Performers (2020, background): “This opens a large volume of new potential applications including: hierarchical attention networks (HANS) (Yang et al., 2016), graph attention networks (Velickovic et al., 2018), image processing (Fu et al., 2019), and reinforcement learning/robotics (Tang et al., 2020).”
- Graph-based, Self-Supervised Program Repair from Diagnostic Feedback (2020, methodology): “We then design a neural net model with a graph-attention mechanism (Velikovi et al., 2018) on the program-feedback graph to model the symbol tracking process described above.”
- Reasoning Over Semantic-Level Graph for Fact Checking (2019, methodology): “We first employ a graph attention mechanism (Veliˇckovi´c et al., 2017) to generate a claim-specific evidence representation for each node in claim-based graph.”
- Neural Execution Engines: Learning to Execute Subroutines (2020, background): “A NEE is essentially a graph attention network [27] that can modify its own graph, resulting in a new decoding mechanism.”
- Knowledge Graph Convolutional Networks for Recommender Systems (2019, background): “xed-size set of neighbors as the support size [7]. Our work can be seen as a non-spectral method for a special type of graphs (i.e., knowledge graph). Our method also connects to PinSage [21] and GAT [15]. But note that both PinSage and GAT are designed for homogeneous graphs. The major difference b”
- Strategies for Pre-training Graph Neural Networks (2019, background): “We also experimented with other popular architectures that are less expressive: GCN (Kipf & Welling, 2016), GAT (Veliˇckovi´c et al., 2019), and GraphSAGE (with mean neighborhood aggregation) (Hamilton et al., 2017b).”
- Graph Convolutional Networks for Text Classification (2018, background): “In addition, some more complex features have been designed, such as n-grams (Wang and Manning 2012) and entities in ontologies (Chenthamarakshan et al. 2011).”
- Fast Graph Representation Learning with PyTorch Geometric (2019, methodology): “Among the experiments, the APPNP operator (Klicpera et al., 2019) generally performs best, with ARMA (Bianchi et al., 2019), SGC (Wu et al., 2019), GCN (Kipf & Welling, 2017) and GAT (Veličković et al., 2018) following closely behind.”
- Auto-GNN: Neural architecture search of graph neural networks (2019, methodology): “Following the model configurations in baselines [10, 32], the training experiments are set up according to transductive learning and inductive learning, respectively.”
- On the Binding Problem in Artificial Neural Networks (2020, background): “…applied for relational reasoning about objects (Zambaldi et al., 2019; van Steenkiste et al., 2020; Goyal et al., 2019; Santoro et al., 2018a), citation networks (Veličković et al., 2018), question answering (Dehghani et al., 2019), and language modeling (Devlin et al., 2019; Brown et al., 2020).”
- Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication (2020, neutral): “A range of relation kernels has been proposed [9,12,21].”
- Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, background): “It is also possible to use graph attention networks (Veličković et al., 2018) for this task (see e.”
- Relational inductive biases, deep learning, and graph networks (2018, methodology): “But various NLNN-compliant models, such as the vertex attention interaction network (Hoshen, 2017) and graph attention network (Veliˇckovi´c et al., 2018), are able to handle explicit edges by effectively setting to zero the weights between nodes which do not share an edge.”
- Causal Interpretability for Machine Learning - Problems, Methods and Evaluation (2020, background): “Attention networks have been successful in various highly-impactful tasks such as graph embedding [99] and machine translation [98, 6].”
- Position-aware Graph Neural Networks (2019, methodology): “…information (Hamilton et al., 2017a); Graph Attention Networks aggregate neighborhood information according to trainable attention weights (Velickovic et al., 2018); Message Passing Neural Networks further incorporate edge information when doing the aggregation (Gilmer et al., 2017);…”
- Unsupervised Inductive Whole-Graph Embedding by Preserving Graph Proximity (2019, methodology): “Examples include GCN as in (Kipf & Welling, 2016b), as well as a series of improvements upon GCN, such as GRAPHSAGE (Hamilton et al., 2017), GAT (Velickovic et al., 2018), GIN (Xu et al., 2019), etc. as mentioned in the main paper.”
- Unsupervised Inductive Graph-Level Representation Learning via Graph-Graph Proximity (2019, methodology): “, 2017], GAT [Velickovic et al., 2018], GIN [Xu et al.”
- Graph Warp Module: an Auxiliary Module for Boosting the Power of Graph Neural Networks (2019, methodology): “Graph Attention Networks (GATs) [28] introduce a self-attention mechanism [27], which is equipped with a trainable set of weights that controls the importance of edges for each node.”
- How Powerful are Graph Neural Networks? (2018, background): “Recently, there has been a surge of interest in Graph Neural Network (GNN) approaches for representation learning of graphs (Li et al., 2016; Hamilton et al., 2017a; Kipf & Welling, 2017; Velickovic et al., 2018; Xu et al., 2018).”