Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
By SHIYANG LI, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, Xifeng Yan. 2019
1,983 citations recorded by the source corpus.
Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. In this paper, we propose to tackle such forecasting problem with Transformer [1]. Although impressed by its performance in our preliminary study, we found its two major weaknesses: (1) locality-agnostics: the point-wise dot-product self-attention in canonical Transformer architecture is insensitive to local context, which can make the model prone to anomalies in time series; (2) memory bottleneck: space complexity of canonical Transformer grows quadratically with sequence length $L$, making directly modeling long time series infeasible. In order to solve these two issues, we first propose convolutional self-attention by producing queries and keys with causal convolution so that local context can be better incorporated into attention mechanism. Then, we propose LogSparse Transformer with only $O(L(\log L)^{2})$ memory cost, improving forecasting accuracy for time series with fine granularity and strong long-term dependencies under constrained memory budget. Our experiments on both synthetic data and real-world datasets show that it compares favorably to the state-of-the-art.
Subfields: Artificial Intelligence, Management Science and Operations Research, Model Architectures, Signal Processing, Time-Series & Forecasting
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Quoc V. Le
- Yoshua Bengio
- Noah A. Smith
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Sergey Levine
In Management Science and Operations Research
In Model Architectures
In Signal Processing
In Time-Series & Forecasting
- Bryan Lim
- Bernie Wang
- Stefan Zohren
- Tim Januschowski
- Jan Gasthaus
- Haixu Wu
- Mingsheng Long
- Jianmin Wang
Citations in the corpus
- Time-series forecasting with deep learning: a survey (2020, background): “Iterative approaches to multi-horizon forecasting typically make use of AR deep learning architectures [37,39,40,53]—producing multi-horizon forecasts by recursively feeding samples of the target into future time steps (figure 2a).”
- Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting (2019, methodology): “Recent papers have also proposed the use of attention-based mechanisms for time-series interpretability [1, 7, 25], with both LSTM-based [33] and transformer-based [25] models.”
- Blockwise Self-Attention for Long Document Understanding (2019, positive): “Most of them focus on the sparsification of attention matrix, such as Star Transformer (Guo et al., 2019), Sparse Transformer (Child et al., 2019), Adaptive Sparse Transformer (Correia et al., 2019; Sukhbaatar et al., 2019), Log-Sparse Transformer (Li et al., 2019), etc.”
- Random Feature Attention (2021, background): “In learning problems where such inductive bias is crucial (Ba et al., 2016; Parmar et al., 2018; Miconi et al., 2018; Li et al., 2019, inter alia ), transformers heavily rely on positional encodings.”
- A Transformer-based Framework for Multivariate Time Series Representation Learning (2020, methodology): “[19] showed superior performance compared to the classical statistical method ARIMA, the recent matrix factorization method TRMF, an RNN-based autoregressive model (DeepAR) and an RNN-based state space model (DeepState) on 4 public forecasting datasets, while Wu et al.”
- Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structures (2022, methodology): “In addition to LSTNet and SCINet, we compare with the following baseline methods for long sequence forecasting: • LogTrans [9].”
- Learning Dynamics and Heterogeneity of Spatial-Temporal Graph Data for Traffic Forecasting (2021, methodology): “To mask future information, following [50] the 1D convolu-tions on the queries and keys are replaced with the causal convolutions [16].”
- Traffic transformer: Capturing the continuity and periodicity of time series for traffic forecasting (2020, background): “It is built entirely upon attention mechanisms, which makes it possible to access any part of a sequence regardless of its distance to the target (Li et al., 2019; Vaswani et al., 2017).”
- ETSformer: Exponential Smoothing Transformers for Time-series Forecasting (2022, methodology): “Owing to their powerful capability to model sequential data, Transformer-based architectures [20, 37, 38, 40, 41] have been actively explored for the time-series forecasting, especially for the more challenging Long Sequence Time-series Forecasting (LSTF) task.”
- Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (2020, methodology): “Due to the missing data (Li et al. 2019), we convert the dataset into hourly consumption of 2 years and set ‘MT 320’ as the target value.”
- TSMixer: An all-MLP Architecture for Time Series Forecasting (2023, background): “In contrast, numerous deep learning models, particularly Transformer-based models, have been proposed due to their capacity to capture both complex temporal patterns and cross-variate dependencies (Gamboa, 2017; Li et al., 2019; Wen et al., 2017; Zhou et al., 2021; Wu et al., 2021; Lim & Zohren, 202”
- Transformers in Time-Series Analysis: A Tutorial (2022, methodology): “Li et al. proposed LogSparse Transformers to overcome memory challenges, thus making Transformers more feasible for time-series data with long-term dependencies [47].”
- ActionFormer: Localizing Moments of Actions with Transformers (2022, methodology): “Adding convolutions before a Transformer network was recently found helpful to better incorporate local context for time series data [32] and to stabilize the training of vision Transformers [72].”
- Are Transformers Effective for Time Series Forecasting? (2022, methodology): “We include six state-of-the-art Transformer-based methods: FEDformer [29], Autoformer [27], Informer [28], Pyraformer [18], LogTrans [16], and Reformer [14].”
- Is Mamba Effective for Time Series Forecasting? (2024, positive): “Many models attempt to reduce the computational complexity of the transformer in TSF by modifying its structure, such as focusing only on a portion of the sequence [28, 71, 31].”
- Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models (2022, methodology): “We compare to LSTnet [27], LSTMa [3], Reformer [25], LogTrans [28], Informer [52] and one of its variants Informer(†) as baselines.”
- SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction (2021, methodology): “Besides, compared with the vanilla Transformer-based methods [18, 21, 42], the newly-proposed Transformer-based forecasting model Autoformer [38] achieves the second best performance in all experimental settings and also surpasses”
- Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “point forecasting methods [140] as well as probabilistic methods, which explicitly model the data distribution using Gaussian copulas [206], GANs [271], or normalizing flows [192].”