DySAT: Deep Neural Representation Learning on Dynamic Graphs via Self-Attention Networks

By Aravind Sankar, Yanhong Wu, Liang Gou, Wei Zhang, Hao Yang. Web Search and Data Mining, 2020

771 citations recorded by the source corpus.

Learning node representations in graphs is important for many applications such as link prediction, node classification, and community detection. Existing graph representation learning methods primarily target static graphs while many real-world graphs evolve over time. Complex time-varying graph structures make it challenging to learn informative node representations over time. We present Dynamic Self-Attention Network (DySAT), a novel neural architecture that learns node representations to capture dynamic graph structural evolution. Specifically, DySAT computes node representations through joint self-attention along the two dimensions of structural neighborhood and temporal dynamics. Compared with state-of-the-art recurrent methods modeling graph evolution, dynamic self-attention is efficient, while achieving consistently superior performance. We conduct link prediction experiments on two graph types: communication networks and bipartite rating networks. Experimental results demonstrate significant performance gains for DySAT over several state-of-the-art graph embedding baselines, in both single and multi-step link prediction tasks. Furthermore, our ablation study validates the effectiveness of jointly modeling structural and temporal self-attention.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/5b3c1f48229f779fc3c9d35ae67bf568cfdc3e2a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistical and Nonlinear Physics

Citations in the corpus

A General Survey on Attention Mechanisms in Deep Learning (2022, background): “For example, self-attention is explored in [82] for the purpose of representation learning in communication networks and rating networks.”
A Comprehensive Survey on Graph Anomaly Detection With Deep Learning (2021, background): “As a popular research topic, deep learning for dynamic graph data mining [182], [183] has shown its effectiveness in supporting many dynamic graph analysis tasks, such as node clustering and graph classiﬁcation [60].”
Incremental Training of Graph Neural Networks on Temporal Graphs under Distribution Shift (2020, background): “DySAT [21] also assumes a ﬁxed vertex set and jointly employs self-attention over structural and temporal dimensions.”
Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks (2021, background): “Most of the above models are not inductive because they associate each node with an onehot identity (or the corresponding row of the adjacency matrix, or a free-trained vector) (Li et al., 2018; Chen et al., 2019; Kumar et al., 2019; Hajiramezanali et al., 2019; Sankar et al., 2020; Manessi et al., ”
Foundations and Modeling of Dynamic Networks Using Dynamic Graph Neural Networks: A Survey (2020, methodology): “If temporal granularity and performance is not a concern then one of the advanced discrete approaches such as DySAT or EvolveGCN will likely be a great fit for most dynamic network problems.”