Long-term Forecasting using Tensor-Train RNNs

By Rose Yu, Stephan Zheng, Anima Anandkumar, Yisong Yue. 2017

146 citations recorded by the source corpus.

We present Tensor-Train RNN (TT-RNN), a novel family of neural sequence architectures for multivariate forecasting in environments with nonlinear dynamics. Long-term forecasting in such systems is highly challenging, since there exist long-term temporal dependencies, higher-order correlations and sensitivity to error propagation. Our proposed tensor recurrent architecture addresses these issues by learning the nonlinear dynamics directly using higher order moments and high-order state transition functions. Furthermore, we decompose the higher-order structure using the tensor-train (TT) decomposition to reduce the number of parameters while preserving the model performance. We theoretically establish the approximation properties of Tensor-Train RNNs for general sequence inputs, and such guarantees are not available for usual RNNs. We also demonstrate significant long-term prediction improvements over general RNN and LSTM architectures on a range of simulated environments with nonlinear dynamics, as well on real-world climate and traffic data.

Subfields: Artificial Intelligence, Computational Mathematics, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mathematics

In Electrical and Electronic Engineering

Citations in the corpus

Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting (2019, methodology): “[3] fuses traditional AR models with RNNs by modeling a probabilistic distribution in an encoder-decoder fashion.”
Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting (2022, methodology): “RNN-based models [33, 36, 23, 29, 30] are proposed for application in an”
TensorLy: Tensor Learning in Python (2016, background): “, 2017), generative adversarial networks (Cao and Zhao, 2017) and sequence models (Yu et al., 2017).”
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (2020, methodology): “However, existing methods are designed under limited problem setting, like predicting 48 points or less (Hochreiter and Schmidhuber 1997; Li et al. 2018; Yu et al. 2017; Liu et al. 2019; Qin et al. 2017; Wen et al. 2017).”
Forecasting Big Time Series: Old and New (2018, background): “Diﬀerent RNN forecasters are introduced [37, 6, 41, 17, 25] and we explain the intuitions behind diﬀerent structures and demonstrate their performances on a variety types of time series.”
Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting (2021, methodology): “Besides, recurrent neural networks (RNNs) models are used to model the temporal dependencies for time series [36, 26, 40, 22].”
Bayesian Temporal Factorization for Multidimensional Time Series Prediction (2019, background): “Third, we would like to integrate recent advances in deep learning to better capture the complex and non-linear dynamics in modern time series data [39], [40], [41], [42], [43], [44], [45], [46].”
Granger Causality: A Review and Recent Advances (2021, methodology): “…(MLPs) (Raissi et al., 2018; Ki¸si, 2004; Billings, 2013) and recurrent neural networks (RNNs) like long-short term memory networks (LSTMs) (Graves, 2012) have shown impressive performance in forecasting multivariate time series given their past (Yu et al., 2017; Zhang, 2003; Li et al., 2017).”
Learning to Reason with Third-Order Tensor Products (2018, background): “, recent work which applies the tensor-train decomposition to RNNs [43, 44].”
Multi-step Deep Autoregressive Forecasting with Latent States (2019, methodology): “Most existing works employ a Recurrent Neural Network (RNN) (Lai et al., 2018; Li et al., 2017; Yu et al., 2017; Flunkert et al., 2017) or a Temporal Convolutional Network (TCN) (Borovykh et al., 2017) to model an AR process.”
Neural Granger Causality (2021, background): “…their time series variants, such as autoregressive multilayer perceptrons (MLPs) [23], [24], [25] and recurrent neural networks (RNNs) like long-short term memory networks (LSTMs) [26] have shown impressive performance in forecasting multivariate time series given their past [27], [28], [29].”
A Unified Framework for Long Range and Cold Start Forecasting of Seasonal Profiles in Time Series (2017, background): “The long-range forecasting problem has traditionally been understudied, though see [1, 2] for recent approaches.”