A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction
By Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, Garrison W. Cottrell. 2017
1,569 citations recorded by the source corpus.
The Nonlinear autoregressive exogenous (NARX) model, which predicts the current value of a time series based upon its previous values as well as the current and past values of multiple driving (exogenous) series, has been studied for decades. Despite the fact that various NARX models have been developed, few of them can capture the long-term temporal dependencies appropriately and select the relevant driving series to make predictions. In this paper, we propose a dual-stage attention-based recurrent neural network (DA-RNN) to address these two issues. In the first stage, we introduce an input attention mechanism to adaptively extract relevant driving series (a.k.a., input features) at each time step by referring to the previous encoder hidden state. In the second stage, we use a temporal attention mechanism to select relevant encoder hidden states across all time steps. With this dual-stage attention scheme, our model can not only make predictions effectively, but can also be easily interpreted. Thorough empirical studies based upon the SML 2010 dataset and the NASDAQ 100 Stock dataset demonstrate that the DA-RNN can outperform state-of-the-art methods for time series prediction.
Subfields: Artificial Intelligence, Management Science and Operations Research, Model Architectures, Signal Processing, Time-Series & Forecasting
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Management Science and Operations Research
In Model Architectures
In Signal Processing
In Time-Series & Forecasting
- Bernie Wang
- Jan Gasthaus
- Tim Januschowski
- Haixu Wu
- Mingsheng Long
- Jianmin Wang
- Syama Sundar Rangapuram
- David Salinas
Citations in the corpus
- N-BEATS: Neural basis expansion analysis for interpretable time series forecasting (2019, background): “A few earlier works explored the combinations of recurrent neural networks with dilation, residual connections and attention [5, 17, 26].”
- A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting (2020, positive): “Each block is a sequence of one to four layers, belonging to one of the three types of dilated LSTMs: standard (Chang et al., 2017), with an attention mechanism (Qin, Song, Chen, Cheng, Jiang, & Cottrell, 2017) and a special residual version (Kim, El-Khamy, & Lee, 2017).”
- Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions (2019, background): “Embedding all the information in a fixed-size vector can result in information loss (Qin et al., 2017).”
- A Review of Deep Learning Models for Time Series Prediction (2019, methodology): “Recently the attention mechanism was modified for use in time series prediction [97], [98].”
- Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting (2023, methodology): “With the development of deep learning, many models have been proposed and achieved superior performances in MTS forecasting (Lea et al., 2017; Qin et al., 2017; Flunkert et al., 2017; Rangapuram et al., 2018; Li et al., 2019a; Wu et al., 2020; Li et al., 2021).”
- A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data (2018, methodology): “Despite the intrinsic unsupervised setting, most of them may still not be able to detect anomalies effectively due to the following reasons: (Hamilton 1994) and LSTM encoder-decoder (Malhotra et al. 2016; Qin et al. 2017), and increase the false positive detections.”
- Anomaly Detection for IoT Time-Series Data: A Survey (2020, methodology): “Attention-based RNNs are employed within an autoencoder in [70] to more accurately predict complex long term patterns within data.”
- Dynamic Graph Convolutional Recurrent Network for Traffic Prediction: Benchmark and Solution (2021, background): “Attention mechanism is also powerful and helpful in sequential dependency modeling, including selfattention [9]–[11], [13] and other variants [4], [20], [24], [25].”
- Deep Learning With Spatiotemporal Attention-Based LSTM for Industrial Soft Sensor Model Development (2020, background): “In recent years, attention mechanisms have performed well for many different tasks like machine translation, image classification, and natural language processing [40], [41].”
- Parallel spatio-temporal attention-based TCN for multivariate time series prediction (2021, methodology): “Our experiments show PSTA-TCN framework has three distinct advantages over the current alternatives: • Speed: PSTA-TCN trains 14 times faster than the current state-of-the-art DSTP [35] and 12 times faster than DSTP’s predecessor DARNN [34].”
- The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches (2018, background): “A dual state attention based RNN is proposed for effective time series prediction [146].”
- Exploring Interpretable LSTM Neural Networks over Multi-Variable Data (2019, methodology): “Moreover, weighting input data by attentions (Xu et al., 2018; Qin et al., 2017; Choi et al., 2016) does not consider the direction of correlation with the target, which could impair the prediction performance.”
- FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting (2022, background): “Since classic models like ARIMA and basic RNN/CNN models perform relatively inferior as shown in (Zhou et al., 2021) and (Wu et al., 2021), we mainly include four state-of-the-art transformer-based models for comparison, i.e., Auto-former (Wu et al., 2021), Informer (Zhou et al., 2021), Log-Trans (L”
- AdaRNN: Adaptive Learning and Forecasting of Time Series (2021, methodology): “These deep methods tackle TS classification or forecasting by leveraging attention mechanisms [23, 33] or tensor factorization [40] for capturing shared information among series.”
- Probabilistic Forecasting with Spline Quantile Function RNNs (2019, background): “…by deep learning-based techniques that can learn complex patterns across time series and make use of rich meta-data without significant manual feature engineering effort (Flunkert et al., 2017; Mukherjee et al., 2018; Qin et al., 2017; Smyl et al., 2018; Bandara et al., 2017; Wen et al., 2017).”