Time-Dependent Representation for Neural Event Sequence Prediction

By Yang Li, Nan Du, Samy Bengio. 2017

64 citations recorded by the source corpus.

Existing sequence prediction methods are mostly concerned with time-independent sequences, in which the actual time span between events is irrelevant and the distance between events is simply the difference between their order positions in the sequence. While this time-independent view of sequences is applicable for data such as natural languages, e.g., dealing with words in a sentence, it is inappropriate and inefficient for many real world events that are observed and collected at unequally spaced points of time as they naturally arise, e.g., when a person goes to a grocery store or makes a phone call. The time span between events can carry important information about the sequence dependence of human behaviors. In this work, we propose a set of methods for using time in sequence prediction. Because neural sequence models such as RNN are more amenable for handling token-like input, we propose two methods for time-dependent event representation, based on the intuition on how time is tokenized in everyday life and previous work on embedding contextualization. We also introduce two methods for using next event duration as regularization for training a sequence prediction model. We discuss these methods based on recurrent neural nets. We evaluate these methods as well as baseline models on five datasets that resemble a variety of sequence prediction tasks. The experiments revealed that the proposed methods offer accuracy gain over baseline models in a range of settings.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

Self-attention with Functional Time Representation Learning (2019, methodology): “Also, detecting interactions between temporal and event contexts is an increasingly important topic in user behavioural modelling [12].”
Relational Representation Learning for Dynamic (Knowledge) Graphs: A Survey (2019, background): “…problems [160, 9, 94, 158, 103, 200], where the duration between any two consecutive items in the sequence is considered equal, and several asynchronous sequence modeling problems [45, 143, 64, 167, 260, 100], RNNs have been the most common choice for extending sequence models to DTDGs and CTDGs.”
Time2Vec: Learning a Vector Representation of Time (2019, background): “There is also a set of techniques that consider time as yet another feature and concatenate time (or some hand designed features of time such as log and/or inverse of delta time) with the input [10, 33, 14, 3, 29, 55, 28, 38].”