Are Transformers Effective for Time Series Forecasting?

By Ailing Zeng, Mu-Hwa Chen, Lei Zhang, Qiang Xu. AAAI Conference on Artificial Intelligence, 2022

4,138 citations recorded by the source corpus.

Recently, there has been a surge of Transformer-based solutions for the long-term time series forecasting (LTSF) task. Despite the growing performance over the past few years, we question the validity of this line of research in this work. Specifically, Transformers is arguably the most successful solution to extract the semantic correlations among the elements in a long sequence. However, in time series modeling, we are to extract the temporal relations in an ordered set of continuous points. While employing positional encoding and using tokens to embed sub-series in Transformers facilitate preserving some ordering information, the nature of the permutation-invariant self-attention mechanism inevitably results in temporal information loss. To validate our claim, we introduce a set of embarrassingly simple one-layer linear models named LTSF-Linear for comparison. Experimental results on nine real-life datasets show that LTSF-Linear surprisingly outperforms existing sophisticated Transformer-based LTSF models in all cases, and often by a large margin. Moreover, we conduct comprehensive empirical studies to explore the impacts of various design elements of LTSF models on their temporal relation extraction capability. We hope this surprising finding opens up new research directions for the LTSF task. We also advocate revisiting the validity of Transformer-based solutions for other time series analysis tasks (e.g., anomaly detection) in the future.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/5f404dbba07619cc7f28d75d03f124a52290046e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting (2023, methodology): “We further compare with two additional concurrent methods which were either not peerreviewed (Grigsby et al., 2022) or were accepted after the submission of this work (Zeng et al., 2023): 1) STformer (Grigsby et al., 2022), a Transformer-based model that directly flattens the multivariate time-serie”
Unified Training of Universal Time Series Forecasting Transformers (2024, methodology): “We report the Mean Squared Error (MSE) and MAE, comparing against six state-of-the-art baselines, iTransformer (Liu et al., 2023b), TimesNet (Wu et al., 2023), PatchTST, Crossformer (Zhang & Yan, 2023), TiDE, DLinear (Zeng et al., 2023), SCINet (Liu et al., 2022), and FEDformer (Zhou et al., 2022b).”
Transformers in Time Series: A Survey (2022, result): “Remarks Note that DLinear [Zeng et al. , 2023] questions the necessity of using Transformers for long-term time series forecasting, and shows that a simpler MLP-based model can achieve better results compared to some Transformer base-lines through empirical studies.”
ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis (2024, methodology): “Since a longer input length indicates more historical information an algorithm can utilize in time series forecasting, a model with strong ability to capture long-term temporal dependency should perform better when input length increases (Zeng et al., 2022; Wang et al., 2023; Nie et al., 2023).”
Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (2024, neutral): “…model sparsification has received relatively less attention, as time series models have traditionally been small in scale, with simple models like DLinear (Zeng et al., 2023) and SparseTSF (Lin et al., 2024) excelling in specific tasks prior to the advent of large-scale, general pre-training.”
TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting (2023, methodology): “The DLinear model [34] also employs a simple linear model after decomposing the time series into trend and seasonality components.”
TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting (2024, methodology): “According to foundation backbones, deep models can be roughly categorized into the following four paradigms: RNN-, CNN-, Transformer-and MLP-based methods.”
One Fits All: Power General Time Series Analysis by Pretrained LM (2023, background): “Zeng et al. (2023) shows that simply repeating the last value in the look-back window can outperform or be comparable to the best results.”
FITS: Modeling Time Series with 10k Parameters (2023, methodology): “…in comparison to state-of-the-art time series forecasting models, including PatchTST (Nie et al., 2023), TimesNet (Wu et al., 2023), FEDFormer (Zhou et al., 2022a) and LTSF-Linear (Zeng et al., 2023), we rerun all the experiment with code and scripts provided by their official implementation 1 .”
A decoder-only foundation model for time-series forecasting (2023, background): “In recent long-term forecasting works (Zeng et al., 2023; Nie et al., 2022; Das et al., 2023) it has been observed that directly predicting the entire forecasting horizon in one shot from a decoder can yield better results than auto-regressive decoding on long horizon benchmarks.”
TSMixer: An all-MLP Architecture for Time Series Forecasting (2023, methodology): “However, Zeng et al. (2023) revealed that this is not always the case – Transformer-based models can indeed be significantly worse than simple univariate temporal linear models on many commonly used forecasting benchmarks.”
Frequency-domain MLPs are More Effective Learners in Time Series Forecasting (2023, methodology): “Therefore, the methods based on multi-layer perceptrons (MLPs) have been recently introduced with simple structure, low complexity, and superior forecasting performance, such as N-BEATS [19], LightTS [20], DLinear [21], etc.”
iTransformer: Inverted Transformers Are Effective for Time Series Forecasting (2023, methodology): “Besides, while series variations can be greatly influenced by the sequence order, permutation-invariant attention mechanisms are improperly adopted on the temporal dimension (Zeng et al., 2023).”
Spatio-Temporal Adaptive Embedding Makes Vanilla Transformer SOTA for Traffic Forecasting (2023, methodology): “Following [38], we randomly shuffle the raw input along temporal axis T .”
Large Language Models Are Zero-Shot Time Series Forecasters (2023, methodology): “Consequently, simple time series methods (e.g. ARIMA [8], and linear models [52]) often outperform deep learning methods on popular benchmarks [24].”
A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (2022, methodology): “Unfortunately, regardless of the complicated design of Transformer-based models, it is shown in the recent paper (Zeng et al., 2022) that a very simple linear model can outperform all of the previous models on a variety of common benchmarks and it challenges the usefulness of Transformer for time…”