Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting

By Haixu Wu, Jiehui Xu, Jianmin Wang, Mingsheng Long. Neural Information Processing Systems, 2021

5,097 citations recorded by the source corpus.

Extending the forecasting time is a critical demand for real applications, such as extreme weather early warning and long-term energy consumption planning. This paper studies the long-term forecasting problem of time series. Prior Transformer-based models adopt various self-attention mechanisms to discover the long-range dependencies. However, intricate temporal patterns of the long-term future prohibit the model from finding reliable dependencies. Also, Transformers have to adopt the sparse versions of point-wise self-attentions for long series efficiency, resulting in the information utilization bottleneck. Going beyond Transformers, we design Autoformer as a novel decomposition architecture with an Auto-Correlation mechanism. We break with the pre-processing convention of series decomposition and renovate it as a basic inner block of deep models. This design empowers Autoformer with progressive decomposition capacities for complex time series. Further, inspired by the stochastic process theory, we design the Auto-Correlation mechanism based on the series periodicity, which conducts the dependencies discovery and representation aggregation at the sub-series level. Auto-Correlation outperforms self-attention in both efficiency and accuracy. In long-term forecasting, Autoformer yields state-of-the-art accuracy, with a 38% relative improvement on six benchmarks, covering five practical applications: energy, traffic, economics, weather and disease. Code is available at this repository: \url{https://github.com/thuml/Autoformer}.

Subfields: Electrical and Electronic Engineering, Management Science and Operations Research, Signal Processing

View Paper: https://www.semanticscholar.org/paper/fc46ccb83dc121c33de7ab6bdedab7d970780b2f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

1. Haixu Wu
2. Liang Sun
3. Tian Zhou
4. Junchi Yan
5. Mingsheng Long
6. Qiang Xu
7. Ailing Zeng
8. Jianmin Wang

In Management Science and Operations Research

1. Junchi Yan
2. Qiang Xu
3. Ailing Zeng
4. Donghao Luo
5. Xue Wang
6. Shiyu Wang
7. Liang Sun
8. Tian Zhou

In Signal Processing

1. Junchi Yan
2. Donghao Luo
3. Xue Wang
4. Qiang Xu
5. Ailing Zeng
6. Shiyu Wang
7. Yunhao Zhang
8. Haixu Wu

Citations in the corpus

• Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structures (2022, methodology): “Informer [27] and Autoformer [23] further explored the potential of Transformer in capturing very long-range dependency of time series.”
• Frequency-domain MLPs are More Effective Learners in Time Series Forecasting (2023, methodology): “For the task of long-term forecasting , we adopt Weather [14], Exchange [10], Traffic [14], Electricity [14], and ETT datasets [13], following previous long time series forecasting works [13, 14, 30, 39].”
• Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (2024, neutral): “Datasets like Weatherbench , CMIP6 , and ERA5 are particularly large, often leading to data imbalance and homogenization.”
• Spatio-Temporal Meta-Graph Learning for Traffic Forecasting (2022, positive): “Third, motivated by (Vaswani et al. 2017), a series of traffic trans-formers (Zheng et al. 2020; Xu et al. 2020) and time series transformers (Li et al. 2019; Zhou et al. 2021; Xu et al. 2021) were proposed to do the long sequence time series modeling.”
• Are Transformers Effective for Time Series Forecasting? (2022, methodology): “We include six state-of-the-art Transformer-based methods: FEDformer [29], Autoformer [27], Informer [28], Pyraformer [18], LogTrans [16], and Reformer [14].”
• ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis (2024, methodology): “…methods and MLP-based models is especially compelling (Nie et al., 2023; Zhang & Yan, 2023; Zhou et al., 2022; Cirstea et al., 2022; Wu et al., 2021; Liu et al., 2021a; Li et al., 2019b; Kitaev et al., 2020; Vaswani et al., 2017) (Li et al., 2023b; Zhang et al., 2022; Zeng et al.,…”
• One Fits All: Power General Time Series Analysis by Pretrained LM (2023, methodology): “…(2017); Devlin et al. (2019); Dosovitskiy et al. (2021); Rao et al. (2021), transformer has been introduced to various time series tasks with promising results Wen et al. (2023), especially for time series forecasting Lim et al. (2021); Zhou et al. (2022, 2021); Wu et al. (2021); Nie et al. (2022).”
• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (2022, methodology): “Autoformer (Wu et al., 2021) uses auto-correlation to get patch level connections, but it is a handcrafted design which doesn’t include all the semantic information within a patch.”
• MICN: Multi-scale Local and Global Context Modeling for Long-term Series Forecasting (2023, methodology): “Like Autoformer (Wu et al., 2021b), previous methods decompose the time series and then take the mean prediction of the trend information, which is then added to the other trend information obtained from the decomposition module in the model.”
• FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting (2022, methodology): “Since classic models such as ARIMA and simple RNN/TCN models perform inferior as shown in Zhou et al. (2021) and Wu et al. (2021), we mainly include five state-of-the-art (SOTA) Transformer-based models, i.e., FEDformer, Autoformer Wu et al. (2021), Informer Zhou et al. (2021), Log-Trans Li et al. (2”
• TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting (2024, methodology): “Firstly, for the series decomposition, Autoformer (Wu et al., 2021) presents the series decomposition block based on moving average to decompose complex temporal variations into seasonal and trend components.”
• PromptCast: A New Prompt-Based Learning Paradigm for Time Series Forecasting (2022, methodology): “As for the hyperparameter searching, the factor c (used in Informer and Autoformer) is the one that could affect the forecasting performance of numerical forecasting methods [11]).”
• Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting (2022, methodology): “Autoformer [37] fuses the decomposition blocks into a canonical structure and develops Auto-Correlation to discover series-wise connections.”
• FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting (2022, methodology): “For better comparison, we follow the experiment settings of Autoformer in (Wu et al., 2021) where the input length is fixed to 96, and the prediction lengths for both training and evaluation are fixed to be 96, 192, 336, and 720, respectively.”