Self-attention with Functional Time Representation Learning

By Da Xu, Chuanwei Ruan, Evren Körpeoglu, Sushant Kumar, Kannan Achan. 2019

135 citations recorded by the source corpus.

Sequential modelling with self-attention has achieved cutting edge performances in natural language processing. With advantages in model flexibility, computation complexity and interpretability, self-attention is gradually becoming a key component in event sequence models. However, like most other sequence models, self-attention does not account for the time span between events and thus captures sequential signals rather than temporal patterns. Without relying on recurrent network structures, self-attention recognizes event orderings via positional encoding. To bridge the gap between modelling time-independent and time-dependent event sequence, we introduce a functional feature map that embeds time span into high-dimensional spaces. By constructing the associated translation-invariant time kernel function, we reveal the functional forms of the feature map under classic functional function analysis results, namely Bochner's Theorem and Mercer's Theorem. We propose several models to learn the functional time representation and the interactions with event representation. These methods are evaluated on real-world datasets under various continuous-time event sequence prediction tasks. The experiments reveal that the proposed methods compare favorably to baseline models while also capture useful time-event interactions.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, methodology): “[42] use random Fourier features (RFF) [34] to approximate stationary kernels with a sinusoidal input mapping and propose techniques to tune the mapping parameters.”
Inductive Representation Learning on Temporal Graphs (2020, background): “The same idea was explored in a concurrent work (Xu et al., 2019a).”
Foundations and Modeling of Dynamic Networks Using Dynamic Graph Neural Networks: A Survey (2020, methodology): “Another example, is the functional time embedding introduced by Xu et al. [114] which converts learning temporal patterns to the kernel learning problem and learns the kernel function.”
Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks (2021, methodology): “For f2(t), we adopt random Fourier features to encode time (Xu et al., 2019; Kazemi et al., 2019) which may approach any positive definite kernels according to the Bochner’s theorem (Bochner, 1992).”