Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting

By Kashif Rasul, Calvin Seward, Ingmar Schuster, Roland Vollgraf. 2021

554 citations recorded by the source corpus.

In this work, we propose \texttt{TimeGrad}, an autoregressive model for multivariate probabilistic time series forecasting which samples from the data distribution at each time step by estimating its gradient. To this end, we use diffusion probabilistic models, a class of latent variable models closely connected to score matching and energy-based methods. Our model learns gradients by optimizing a variational bound on the data likelihood and at inference time converts white noise into a sample of the distribution of interest through a Markov chain using Langevin sampling. We demonstrate experimentally that the proposed autoregressive denoising diffusion model is the new state-of-the-art multivariate probabilistic forecasting method on real-world data sets with thousands of correlated dimensions. We hope that this method is a useful tool for practitioners and lays the foundation for future research in this area.

Subfields: Artificial Intelligence, Economics and Econometrics, Generative Models, Probabilistic & Bayesian Methods, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Generative Models

In Probabilistic & Bayesian Methods

In Signal Processing

Citations in the corpus

Noise Estimation for Generative Diffusion Models (2021, methodology): “The recent TimeGrad model [24] is a diffusion process for probabilistic time series forecasting, which was shown empirically to outperform Transformers [34] and LSTMs [8] on some datasets.”
Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (2022, neutral): “, 2021) and time series forecasting (Rasul et al., 2021).”
CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation (2021, methodology): “We exclude TimeGrad [25] from the baselines, as they did not report these metrics.”
Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models (2022, background): “, 2021), 3d point cloud generation (Luo & Hu, 2021) and time series forecasting (Rasul et al., 2021).”
Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion (2024, unknown): “[49] modifies auto-regressive models to denoise the next token conditioned on previous tokens.”
Deep Learning for Time Series Forecasting: Tutorial and Literature Survey (2020, methodology): “TimeGrad [150] RNN Prob ≥ 1 NLL RMSE electricity, traffic, solar, exchange rate, taxi, wikipedia Multivariate forecasting using diffusion models.”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “TimeGrad [191] presents an autoregressive model for forecasting multivariate probabilistic time series, which samples from the data distribution at each time step through estimating its gradient.”
Implicit Diffusion Models for Continuous Super-Resolution (2023, unknown): “Recently, Diffusion Probabilistic Models (DMs) [12] have shown state-of-the-art results in image and speech synthesis [5], and time series forecasting [32], for example.”
Executing your Commands via Motion Diffusion in Latent Space (2022, background): “Nevertheless, the raw motion sequences are somewhat time-axis redundant, and diffusion models in raw sequential data [54, 22, 35] usually require exhausting computational overhead in both the training and inference phase, which is inefficient.”
Diffusion Probabilistic Modeling for Video Generation (2022, methodology): “Equations 7 and 8 generalize and improve the previously proposed TimeGrad (Rasul et al., 2021) method.”