Self-Attention Graph Pooling

By Junhyun Lee, Inyeop Lee, Jaewoo Kang. 2019

1,340 citations recorded by the source corpus.

Advanced methods of applying deep learning to structured data such as graphs have been proposed in recent years. In particular, studies have focused on generalizing convolutional neural networks to graph data, which includes redefining the convolution and the downsampling (pooling) operations for graphs. The method of generalizing the convolution operation to graphs has been proven to improve performance and is widely used. However, the method of applying downsampling to graphs is still difficult to perform and has room for improvement. In this paper, we propose a graph pooling method based on self-attention. Self-attention using graph convolution allows our pooling method to consider both node features and graph topology. To ensure a fair comparison, the same training procedures and model architectures were used for the existing pooling methods and our method. The experimental results demonstrate that our method achieves superior graph classification performance on the benchmark datasets using a reasonable number of parameters.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “Compared to GCNs, both the graph structure and weights are shared across layers (iterations), and the messages (corresponding to the incoming relations) are typically implemented as a pairwise non-linear function of both adjacent node representations.”
A Comprehensive Survey on Graph Neural Networks (2019, methodology): “Most recently, the SAGPool [102] approach is proposed, which considers both node features and graph topology and learns the pooling in a self-attention manner.”
Simple and Deep Graph Convolutional Networks (2020, background): “(Gao & Ji, 2019; Lee et al., 2019) devote to extend pooling operations to graph neural network.”
Towards Deeper Graph Neural Networks (2020, background): “Great successes have been achieved for many applications, such as node classification [7, 9, 11, 12, 21, 30, 31, 33], graph classification [6, 8, 14, 18, 32, 34, 35, 38] and link prediction [1, 36, 37].”
Sequential Recommendation with Graph Neural Networks (2021, methodology): “In this section, we use the graph pooling method [17, 22, 37] to further extract the fused information.”
Graph Neural Networks: A Review of Methods and Applications (2018, methodology): “SAGPool (Lee et al., 2019) is also proposed to use features and topology jointly to learn graph representations.”
Spectral Clustering with Graph Neural Networks for Graph Pooling (2019, methodology): “The approach dubbed Top-K pooling (Hongyang Gao, 2019; Lee et al., 2019), learns a projection vector that is applied to each node feature to obtain a score.”
Visual Semantics Allow for Textual Reasoning Better in Scene Text Recognition (2021, neutral): “Next, we use a readout layer like (Lee, Lee, and Kang 2019) to aggregate node features to a ﬁxed-size representation.”
Pooling Architecture Search for Graph Classification (2021, methodology): “Following the existing pooling architectures [22, 47], 2-layer backbone is chosen in this paper for all NAS baselines and PAS, and more layers can be trivially added along with the increasing number of nodes and edges on new graphs.”
AASIST: Audio Anti-Spoofing Using Integrated Spectro-Temporal Graph Attention Networks (2021, background): “Various graph pooling layers have been proposed to effectively scale down the graph [24,25].”
Representing Long-Range Context for Graph Neural Networks with Global Attention (2022, methodology): “We ﬁnd that this approach outperforms both non-learned readout methods like global pooling as well as learned aggregation methods like graph-speciﬁc pooling methods [37, 20] and “virtual node” approaches.”
Heterogeneous Graph Structure Learning for Graph Neural Networks (2021, methodology): “…graph data, has drawn increasing attention and is widely applied to node classiﬁcation (Kipf and Welling 2017; Velickovic et al. 2018; Xu et al. 2019), graph classiﬁcation (Duvenaud et al. 2015; Lee, Lee, and Kang 2019), and recommendation (Ying et al. 2018; Fan et al. 2019b; Wang et al. 2019a).”
A Gentle Introduction to Deep Learning for Graphs (2019, methodology): “- Adaptive - - DGCNN [125] ✗ Topological ✗ ✗ GAT [102] ✗ ✗ ✓ ✗ R-GCN [90] Discrete ✗ ✗ ✗ GMNN [83] - - - - DGI [103] ✗ ✗ ✗ ✓ GIN [113] ✗ ✗ ✗ ✗ FDGNN [36] ✗ ✗ ✗ ✓ NMFPool [2] - Topological - - SAGPool [62] - Adaptive - - Top-k Pool [37] - Adaptive - - Table 2: Here we recap the main properties of DGN”
ASAP: Adaptive Structure Aware Pooling for Learning Hierarchical Graph Representations (2019, methodology): “Recently, SAG-Pool (Lee, Lee, and Kang 2019), a TopK based architecture, has been proposed which leverages self-attention network to learn the node scores.”
Graph Clustering with Graph Neural Networks (2020, methodology): “As graphs in the real-world vary in size and sparsity, methods cannot use a O(n2) link prediction objectives, like DiffPool [66], or compute A2, like top-k pooling methods [20, 31].”
Probabilistic Dual Network Architecture Search on Graphs (2020, background): “The ﬁrst pooling layer is a self-attention pooling layer (Lee et al., 2019) while the last layer is a global average pooling.”
XGNN: Towards Model-Level Explanations of Graph Neural Networks (2020, background): “In addition, extensive efforts have been made towards different graph operations, such as graph convolution [13, 16, 19], graph pooling [20, 44], and graph attention [10, 36, 37].”
Discovering Invariant Rationales for Graph Neural Networks (2022, methodology): “We summarize two classes of the existing methods to build deep interpretable GNNs, (i) Attention (Vaswani et al., 2017; Veli ˇ ckovi ´ c et al., 2018), which can be broadly interpreted as importance weights on representations.”