Temporal pattern attention for multivariate time series forecasting

By Shun-Yao Shih, Fan-Keng Sun, Hung-yi Lee. Machine-mediated learning, 2018

907 citations recorded by the source corpus.

Forecasting of multivariate time series data, for instance the prediction of electricity consumption, solar power production, and polyphonic piano pieces, has numerous valuable applications. However, complex and non-linear interdependencies between time steps and series complicate this task. To obtain accurate prediction, it is crucial to model long-term dependency in time series data, which can be achieved by recurrent neural networks (RNNs) with an attention mechanism. The typical attention mechanism reviews the information at each previous time step and selects relevant information to help generate the outputs; however, it fails to capture temporal patterns across multiple time steps. In this paper, we propose using a set of filters to extract time-invariant temporal patterns, similar to transforming time series data into its “frequency domain”. Then we propose a novel attention mechanism to select relevant time series, and use its frequency domain information for multivariate forecasting. We apply the proposed model on several real-world tasks and achieve state-of-the-art performance in almost all of cases. Our source code is available at https://github.com/gantheory/TPA-LSTM.

Subfields: Management Science and Operations Research, Signal Processing

View Paper: https://www.semanticscholar.org/paper/5d1d53c671b20db116ba8c91c6446bb4757614da

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

1. Qiang Xu
2. Ailing Zeng
3. Haixu Wu
4. Qingsong Wen
5. Mingsheng Long
6. Jianmin Wang
7. Mu-Hwa Chen
8. Xiaojun Chang

In Signal Processing

1. Qiang Xu
2. Ailing Zeng
3. Haixu Wu
4. Zhijian Xu
5. Mingsheng Long
6. Qingsong Wen
7. Jianmin Wang
8. Mu-Hwa Chen

Citations in the corpus

• Spatio-Temporal Meta-Graph Learning for Traffic Forecasting (2022, neutral): “…superior performance in traffic modeling (Ma et al. 2015a; Lv et al. 2018; Li et al. 2018; Zhao et al. 2019; Wang et al. 2020; Bai et al. 2020; Ye et al. 2021; Shang, Chen, and Bi 2021; Lee et al. 2022) as well as multivariate time series forecasting (Lai et al. 2018; Shih, Sun, and Lee 2019).”
• Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structures (2022, methodology): “LSTNet [8] and TPR-LSTM [17] both employ the convolutional neural networks, the recurrent neural networks, and the attentionmechanism [22] tomodel short-term local dependencies among different time series and long-term temporal dependencies.”
• A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection (2023, background): “…in forecasting time series by capturing nonlinear temporal and spatial patterns more effectively than the linear counterpart [23], many of these approaches, such as LSTNet [111] and TPA-LSTM [112], over-look and implicitly model the rich underlying dynamic spatial correlations betweentimeseries.”
• Discrete Graph Structure Learning for Forecasting Multiple Time Series (2021, methodology): “Although these models are more heavily used for sequence data under the context of natural language processing, they are readily applicable for time series as well (Shih et al., 2019; Li et al., 2019).”
• Graph Neural Networks for Intelligent Transportation Systems: A Survey (2023, methodology): “They compared their model with baselines for both single-step (autoregressive, VAR-MLP [38], Gaussian Process [39], RNN-GRU, LSTNet [40], TPA-LSTM [41]) and multi-step (DCRNN, STGCN, Graph WaveNet, ST-MetaNet, GMAN, and MRA-BGCN [42]) forecasting, and reported the overall competence of their…”
• SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction (2021, methodology): “As can be seen in Table 2, the proposed SCINet outperforms existing RNN/TCN-based (LSTNet [19], TPA-LSTM [34], TCN [4], TCN † ) and Transformer-based [38, 42, 37] TSF solutions in most cases, especially for the Solar-Energy and Exchange-Rate datasets.”
• Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting (2021, background): “DeepAR [34] combines autoregressive methods and RNNs to model the probabilistic distribution of future series.”
• Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks (2020, methodology): “• TPA-LSTM: An attention-recurrent neural network [19].”