Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting

By SHIYANG LI, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, Xifeng Yan. 2019

1,983 citations recorded by the source corpus.

Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. In this paper, we propose to tackle such forecasting problem with Transformer [1]. Although impressed by its performance in our preliminary study, we found its two major weaknesses: (1) locality-agnostics: the point-wise dot-product self-attention in canonical Transformer architecture is insensitive to local context, which can make the model prone to anomalies in time series; (2) memory bottleneck: space complexity of canonical Transformer grows quadratically with sequence length $L$, making directly modeling long time series infeasible. In order to solve these two issues, we first propose convolutional self-attention by producing queries and keys with causal convolution so that local context can be better incorporated into attention mechanism. Then, we propose LogSparse Transformer with only $O(L(\log L)^{2})$ memory cost, improving forecasting accuracy for time series with fine granularity and strong long-term dependencies under constrained memory budget. Our experiments on both synthetic data and real-world datasets show that it compares favorably to the state-of-the-art.

Subfields: Artificial Intelligence, Management Science and Operations Research, Model Architectures, Signal Processing, Time-Series & Forecasting

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Quoc V. Le
  3. Yoshua Bengio
  4. Noah A. Smith
  5. Jianfeng Gao
  6. Percy Liang
  7. Luke S. Zettlemoyer
  8. Sergey Levine

In Management Science and Operations Research

  1. Haixu Wu
  2. Mingsheng Long
  3. Jianmin Wang
  4. Bryan Lim
  5. Junchi Yan
  6. Bernie Wang
  7. Qiang Xu
  8. Ailing Zeng

In Model Architectures

  1. Noah A. Smith
  2. Quoc V. Le
  3. Hanxiao Liu
  4. Bryan Lim
  5. Zihang Dai
  6. Dani Yogatama
  7. Mingsheng Long
  8. Jianmin Wang

In Signal Processing

  1. Haixu Wu
  2. Mingsheng Long
  3. Jianmin Wang
  4. Junchi Yan
  5. Qiang Xu
  6. Ailing Zeng
  7. Yunhao Zhang
  8. Bernie Wang

In Time-Series & Forecasting

  1. Bryan Lim
  2. Bernie Wang
  3. Stefan Zohren
  4. Tim Januschowski
  5. Jan Gasthaus
  6. Haixu Wu
  7. Mingsheng Long
  8. Jianmin Wang

Citations in the corpus