Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks

By Guokun Lai, Wei-Cheng Chang, Yiming Yang, Hanxiao Liu. 2017

2,865 citations recorded by the source corpus.

Multivariate time series forecasting is an important machine learning problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. Temporal data arise in these real-world applications often involves a mixture of long-term and short-term patterns, for which traditional approaches such as Autoregressive models and Gaussian Process may fail. In this paper, we proposed a novel deep learning framework, namely Long- and Short-term Time-series network (LSTNet), to address this open challenge. LSTNet uses the Convolution Neural Network (CNN) and the Recurrent Neural Network (RNN) to extract short-term local dependency patterns among variables and to discover long-term patterns for time series trends. Furthermore, we leverage traditional autoregressive model to tackle the scale insensitive problem of the neural network model. In our evaluation on real-world data with complex mixtures of repetitive patterns, LSTNet achieved significant performance improvements over that of several state-of-the-art baseline methods. All the data and experiment codes are available online.

Subfields: Management Science and Operations Research, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Signal Processing

Citations in the corpus

High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes (2019, methodology): “Similar sequence modeling techniques have also been used in the context of forecasting to make probabilistic predictions for collections of real or integer-valued time series [26, 36, 16].”
Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting (2019, methodology): “[3] fuses traditional AR models with RNNs by modeling a probabilistic distribution in an encoder-decoder fashion.”
Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting (2019, methodology): “There has been a concurrent work [25], where an RNN has been used to evolve a global state common to all time-series.”
ETSformer: Exponential Smoothing Transformers for Time-series Forecasting (2022, methodology): “For the multivariate benchmark, baselines include recently proposed time-series/efﬁcient Transform-ers – Autoformer, Informer, LogTrans, and Reformer [16], and RNN variants – LSTnet [18], and LSTM [11].”
MICN: Multi-scale Local and Global Context Modeling for Long-term Series Forecasting (2023, neutral): “Baselines We include four transformer-based models: FEDformer (Zhou et al., 2022), Autoformer (Wu et al., 2021b), Informer (Zhou et al., 2021), LogTrans (Li et al., 2019b), two RNN-based models: LSTM (Hochreiter & Schmidhuber, 1997), LSTNet (Lai et al., 2018b) and CNN-based model TCN (Bai et al., 20”
TS2Vec: Towards Universal Representation of Time Series (2021, methodology): “LSTnet (Lai et al. 2018): LSTnet combines CNNs and RNNs to incorporate both short-term local dependencies and long-term trends.”
FourierGNN: Rethinking Multivariate Time Series Forecasting from a Pure Graph Perspective (2023, neutral): “…a comprehensive comparison of the forecasting performance between our FourierGNN and several representative and state-of-the-art (SOTA) models on the seven datasets, including classic method VAR [32], deep learning-based models such as SFM [24], LSTNet [33], TCN [8], DeepGLO [34], and CoST [36].”
Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting (2021, methodology): “Recent deep forecasting models [48, 23, 26, 34, 29, 35, 25, 41] have achieved great progress, especially the Transformer-based models.”
Deep Learning on Traffic Prediction: Methods, Analysis, and Future Directions (2021, neutral): “ep learning model, which can model the latent 7 TABLE III TEMPORAL DEPENDENCY MODELING. Category Approach CNN [44], [48], [72], [76], [82], [83], [85], [93] RNN [45]–[47], [51]–[53], [61], [62], [64]–[66], [68], [70], [71], [73]–[75], [77], [78], [80], [84], [86], [88]–[91], [94], [95], [98]–[105] G”
Multi-step Deep Autoregressive Forecasting with Latent States (2019, methodology): “In our experiments, we choose to use GRU cells (Cho et al., 2014) to instantiate the RNN. Note that we also use covariates as an additional input associated with the observations to the TCN module.”
SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction (2021, methodology): “Model SCINet Autoformer [40] Informer [42] Transformer [37] *TCN [4] *TCN† LSTNet [19] TPA-LSTM [34] IMP Metric τ RSE CORR RSE CORR RSE CORR RSE CORR RSE CORR RSE CORR RSE CORR RSE CORR RSE 3 0.”
Is Mamba Effective for Time Series Forecasting? (2024, neutral): “In time series forecasting tasks, the model receives input as a history sequence . and then uses the information to predict a future sequence 𝑈 𝑜𝑢𝑡 = [ 𝑢 𝐿 +1 , 𝑢 𝐿 +2 , … , 𝑢 𝐿 + 𝑇 ] ∈ ℝ 𝑇 × 𝑉 .”
Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting (2021, methodology): “The other method of a short-cut connection from the input to the output of the function approximator, as done in the multivariate point forecasting method LSTNet (Lai et al., 2018), is not applicable here.”
A Survey on Anomaly Detection for Technical Systems using LSTM Networks (2021, background): “Regarding time-variant dynamics, it can be distinguished between stationary and non-stationary [10] or short-term and long-term behavior [11].”
One Fits All: Power General Time Series Analysis by Pretrained LM (2023, methodology): “XGBoost Chen & Guestrin (2016), Rocket Dempster et al. (2020), LSTNet Lai et al. (2018), LSSL Gu et al. (2021), Pyraformer Liu et al. (2021), TCN Franceschi et al. (2019) and Flowformer Huang et al. (2022) are used for classification.”
Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting (2022, background): “Time series often exhibit both short-term and long-term repeating patterns (Lai et al., 2018), and taking them into account is the key to accurate prediction.”
Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models (2022, background): “The original data set contains the solar power production records in the year 2006, sampled every 10 minutes from 137 photovoltaic power plants in Alabama State (Lai et al., 2018).”