Transformer Hawkes Process

By Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, Hongyuan Zha. International Conference on Machine Learning, 2020

399 citations recorded by the source corpus.

Modern data acquisition routinely produce massive amounts of event sequence data in various domains, such as social media, healthcare, and financial markets. These data often exhibit complicated short-term and long-term temporal dependencies. However, most of the existing recurrent neural network based point process models fail to capture such dependencies, and yield unreliable prediction performance. To address this issue, we propose a Transformer Hawkes Process (THP) model, which leverages the self-attention mechanism to capture long-term dependencies and meanwhile enjoys computational efficiency. Numerical experiments on various datasets show that THP outperforms existing models in terms of both likelihood and event prediction accuracy by a notable margin. Moreover, THP is quite general and can incorporate additional structural knowledge. We provide a concrete example, where THP achieves improved prediction performance for learning multiple point processes when incorporating their relational information.

Subfields: Applied Mathematics, Geometry and Topology, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/c6dd45c1986f3577336bc9aa596cbd23d610d904

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Mathematics

1. Yang Song
2. Stefano Ermon
3. Junchi Yan
4. Jiaming Song

In Geometry and Topology

1. Stefano Ermon
2. Junchi Yan
3. Jiaming Song

In Molecular Biology

1. Yang Song
2. Stefano Ermon
3. Junchi Yan
4. Jiaming Song
5. William L. Hamilton

Citations in the corpus

• Transformers in Time Series: A Survey (2022, methodology): “Self-attentive Hawkes process (SAHP) [Zhang et al. , 2020] and Transformer Hawkes process (THP) [Zuo et al. , 2020] adopt Transformer encoder architecture to summarize the influence of historical events and compute the intensity function for event prediction.”
• Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting (2022, methodology): “We chose the MSE loss and the log-likelihood (Zuo et al., 2020) as our loss functions.”
• CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation (2021, methodology): “Following previous studies [29, 30], we use 128-dimensions temporal embedding.”