Attention Models in Graphs

By John Boaz Lee, Ryan A. Rossi, Sungchul Kim, Nesreen K. Ahmed, Eunyee Koh. 2018

175 citations recorded by the source corpus.

Graph-structured data arise naturally in many different application domains. By representing data as graphs, we can capture entities (i.e., nodes) as well as their relationships (i.e., edges) with each other. Many useful insights can be derived from graph-structured data as demonstrated by an ever-growing body of work focused on graph mining. However, in the real-world, graphs can be both large—with many complex patterns—and noisy, which can pose a problem for effective graph mining. An effective way to deal with this issue is to incorporate “attention” into graph mining solutions. An attention mechanism allows a method to focus on task-relevant parts of the graph, helping it to make better decisions. In this work, we conduct a comprehensive and focused survey of the literature on the emerging field of graph attention models. We introduce three intuitive taxonomies to group existing work. These are based on problem setting (type of input and output), the type of attention mechanism used, and the task (e.g., graph classification, link prediction). We motivate our taxonomies through detailed examples and use each to survey competing approaches from a unique standpoint. Finally, we highlight several challenges in the area and discuss promising directions for future work.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Auto-GNN: Neural architecture search of graph neural networks (2019, background): “• Attention function: The real-world graph-structured data could be both complex and noisy [26], which may lead to the inefficient information aggregation.”
Simplifying Graph Convolutional Networks (2019, background): “We refer the readers to Lee et al. (2018) for further comparison.”
Attention in Natural Language Processing (2019, unknown): “[130].”
Relational Graph Attention Networks (2018, background): “A recent approach that began with Graph Attention Networks (GATs), applied attention mechanisms to graphs, and does not share these limitations (Veličković et al., 2017; Gong and Cheng, 2018; Zhang et al., 2018; Monti et al., 2018; Lee et al., 2018).”
Visual Attention Methods in Deep Learning: An In-Depth Survey (2022, background): “Recently, researchers surveyed application-specific attention techniques with an emphasis on NLP-based [65], transformer-based [46, 85], and graph-based approaches [97].”
Pyramid Graph Networks With Connection Attentions for Region-Based One-Shot Semantic Segmentation (2019, background): “Incorporating attention mechanism to effectively handle graph-like data has become popular in recent years [17, 12, 23, 22, 4].”
Hypergraph Convolution and Hypergraph Attention (2019, background): “Hypergraph convolution has an innate attentional mechanism (Lee et al., 2018; Velickovic et al., 2018).”
How Attentive are Graph Attention Networks? (2021, background): “…(Zhang et al., 2018; Thekumparampil et al., 2018; Gao and Ji, 2019; Lukovnikov and Fischer, 2021; Shi et al., 2020; Dwivedi and Bresson, 2020; Busbridge et al., 2019; Rong et al., 2020a; Veličković et al., 2020), and Lee et al. (2018) conducted an extensive survey of attention types in GNNs.”
An Attentive Survey of Attention Models (2019, background): “Related surveys: There have been a few surveys on attention focusing on Computer Vision [Wang and Tax, 2016], and graphs [Lee et al., 2018].”
Attention mechanism in neural networks: where it comes and where it goes (2022, background): “Different from the similar studies, BIGBIRD performs well for genomics data alongside NLP tasks such as question answering.”
Graph convolutional networks: a comprehensive review (2019, background): “graph neural networks with a special focus on graph attention networks [31].”