Attention-based Graph Neural Network for Semi-supervised Learning

By Kiran Koshy Thekumparampil, Chong Wang, Sewoong Oh, Li-Jia Li. 2018

386 citations recorded by the source corpus.

Recently popularized graph neural networks achieve the state-of-the-art accuracy on a number of standard benchmark datasets for graph-based semi-supervised learning, improving significantly over existing approaches. These architectures alternate between a propagation layer that aggregates the hidden states of the local neighborhood and a fully-connected layer. Perhaps surprisingly, we show that a linear model, that removes all the intermediate fully-connected layers, is still able to achieve a performance comparable to the state-of-the-art models. This significantly reduces the number of parameters, which is critical for semi-supervised learning where number of labeled examples are small. This in turn allows a room for designing more innovative propagation layers. Based on this insight, we propose a novel graph neural network that removes all the intermediate fully-connected layers, and replaces the propagation layers with attention mechanisms that respect the structure of the graph. The attention mechanism allows us to learn a dynamic and adaptive local summary of the neighborhood to achieve more accurate predictions. In a number of experiments on benchmark citation networks datasets, we demonstrate that our approach outperforms competing methods. By examining the attention weights among neighbors, we show that our model provides some interesting insights on how neighbors influence each other.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Graph Learning

Citations in the corpus

Fast Graph Representation Learning with PyTorch Geometric (2019, background): “, 2018) and AGNN (Thekumparampil et al., 2018), the Graph Isomorphism Network (GIN) fromXu et al.”
Simplifying Graph Convolutional Networks (2019, background): “Recently, Thekumparampil et al. (2018) proposed a novel attention-based GCN model for citation networks.”
PacGAN: The Power of Two Samples in Generative Adversarial Networks (2017, background): “, graph neural networks [43], [44], [45] or deep”
Simple and Deep Graph Convolutional Networks (2020, background): “Another line of works focuses on the attention-based GCN model (Veličković et al., 2018; Thekumparampil et al., 2018; Zhang et al., 2018), which learn the edge weights at each layer based on node features.”
Pyramid Graph Networks With Connection Attentions for Region-Based One-Shot Semantic Segmentation (2019, background): “Incorporating attention mechanism to effectively handle graph-like data has become popular in recent years [17, 12, 23, 22, 4].”
A General Survey on Attention Mechanisms in Deep Learning (2022, background): “In comparison to the total amount of research that has been done on attention models, research on attention models for semi-supervised learning [123], [124] or unsupervised learning [125], [126] has received limited attention and has only become active recently.”
Robust Graph Convolutional Networks Against Adversarial Attacks (2019, background): “Further improvements include adding an attention mechanism to assign different weights in aggregating node neighborhoods [28, 30, 31, 34], adding residual and jumping connections [32], sampling to improve efficiency [4, 5, 14, 33] considering edge attributes [15, 23, 26], disentangling node represen”
How Attentive are Graph Attention Networks? (2021, methodology): “Transformers and GNNs Many works employed GNNs with Transformer-style attention (dotproduct) (Zhang et al., 2018; Thekumparampil et al., 2018; Gao and Ji, 2019; Lukovnikov and Fischer, 2021; Shi et al., 2020; Dwivedi and Bresson, 2020; Busbridge et al., 2019; Rong et al., 2020a; Veličković et…”
How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision (2022, background): “, 2017), variants of attention in graph neural networks (Thekumparampil et al., 2018; Zhang et al., 2018; Wang et al., 2019a; Gao & Ji, 2019; Zhang et al., 2020; Hou et al., 2020) are trained without direct supervision.”
Graph Neural Networks with Heterophily (2020, background): “2018) and AGNN (Thekumparampil et al. 2018) introduced an edge-level attention mechanism; MixHop (Abu-El-Haija et al.”
Label Efficient Semi-Supervised Learning via Graph Filtering (2019, background): “Other related works include GraphSAGE [25], graph attention networks [51], attention-based graph neural network [47], graph partition neural networks [36], FastGCN [16], dual graph convolutional neural network [65], stochastic GCN [17], Bayesian GCN [58], deep graph infomax [50], LanczosNet [37], et”
XGNN: Towards Model-Level Explanations of Graph Neural Networks (2020, background): “In addition, extensive efforts have been made towards different graph operations, such as graph convolution [13, 16, 19], graph pooling [20, 44], and graph attention [10, 36, 37].”
Unifying Graph Convolutional Neural Networks and Label Propagation (2020, methodology): “Attention Mechanism on Graphs Our method is also conceptually connected to attention mechanism on graphs (Veličković et al., 2018; Thekumparampil et al., 2018; Zhang et al., 2018; Liu et al., 2019b), in which an attention weight αij is learned between node vi and vj .”
SimGNN: A Neural Network Approach to Fast Graph Similarity Computation (2018, background): “[41, 42] assign an attention weight to each neighbor of a node for node-level tasks.”
Joint Object Detection and Multi-Object Tracking with Graph Neural Networks (2021, methodology): “Note that we use AGNNConv [59] in this ablation study and we do not explore more than 3 GNN layers as GPU memory is not enough to fit our model.”