A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

By Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, Jayant R. Kalagnanam. International Conference on Learning Representations, 2022

4,010 citations recorded by the source corpus.

We propose an efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. It is based on two key components: (i) segmentation of time series into subseries-level patches which are served as input tokens to Transformer; (ii) channel-independence where each channel contains a single univariate time series that shares the same embedding and Transformer weights across all the series. Patching design naturally has three-fold benefit: local semantic information is retained in the embedding; computation and memory usage of the attention maps are quadratically reduced given the same look-back window; and the model can attend longer history. Our channel-independent patch time series Transformer (PatchTST) can improve the long-term forecasting accuracy significantly when compared with that of SOTA Transformer-based models. We also apply our model to self-supervised pre-training tasks and attain excellent fine-tuning performance, which outperforms supervised training on large datasets. Transferring of masked pre-trained representation on one dataset to others also produces SOTA forecasting accuracy. Code is available at: https://github.com/yuqinie98/PatchTST.

Subfields: Artificial Intelligence, Management Science and Operations Research, Signal Processing

View Paper: https://www.semanticscholar.org/paper/dad15404d372a23b4b3bf9a63b3124693df3c85e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Signal Processing

Citations in the corpus

One Fits All: Power General Time Series Analysis by Pretrained LM (2023, methodology): “…(2017); Devlin et al. (2019); Dosovitskiy et al. (2021); Rao et al. (2021), transformer has been introduced to various time series tasks with promising results Wen et al. (2023), especially for time series forecasting Lim et al. (2021); Zhou et al. (2022, 2021); Wu et al. (2021); Nie et al. (2022).”
Frequency-domain MLPs are More Effective Learners in Time Series Forecasting (2023, methodology): “A potential explanation is that the channel independent strategy [39] brings more benefit to forecasting.”
TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting (2024, background): “…al., 2023a; Hewage et al., 2020), RNN (Lai et al., 2018; Qin et al., 2017; Salinas et al., 2020), Transformer (Vaswani et al., 2017; Zhou et al., 2021; Wu et al., 2021; Zhou et al., 2022b; Nie et al., 2023) and MLP (Zeng et al., 2023; Zhang et al., 2022; Oreshkin et al., 2019; Challu et al., 2023).”
Long-term Forecasting with TiDE: Time-series Dense Encoder (2023, methodology): “Our model is applied in a channel independent manner (the term was used in (Nie et al., 2022)) i.e the input to the model is the past and covariates of one time-series at a time y 1: L , a ( i ) and it maps it to the prediction of that time-series ˆ y L +1: L + H .”
Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (2023, methodology): “For example, PatchTST (Nie et al., 2023) Although these methods enhance efficiency and accuracy compared to vanilla Transformer, they are mostly designed and optimized for narrow prediction tasks within specific domains.”
Unified Training of Universal Time Series Forecasting Transformers (2024, methodology): “While considering each variate of a multivariate time series independently (Nie et al., 2023; Ekambaram et al., 2023) can sidestep this problem, we expect a universal model to be sufficiently flexible to consider multivariate interactions and take into account exogenous covariates.”
TSMixer: An all-MLP Architecture for Time Series Forecasting (2023, methodology): “As noted previously (Nie et al., 2022; Zeng et al., 2023), it is intuitively expected that multivariate models should be able to leverage cross-variate information to achieve better performance than univariate models.”
FITS: Modeling Time Series with 10k Parameters (2023, methodology): “To effectively harness this data, recent research has given rise to sophisticated models and methodologies (Zhou et al., 2021; Liu et al., 2022a; Zeng et al., 2023; Nie et al., 2023; Zhang et al., 2022).”
ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis (2024, methodology): “Since a longer input length indicates more historical information an algorithm can utilize in time series forecasting, a model with strong ability to capture long-term temporal dependency should perform better when input length increases (Zeng et al., 2022; Wang et al., 2023; Nie et al., 2023).”
TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting (2023, methodology): “We categorize SOTA forecasting benchmarks into the following categories: (i) Standard Transformers: FEDformer [38], Autoformer[30] and Informer [37], (ii) Patch Transformers: PatchTST [22] and CrossFormer [3], (iii) MLPs and Non-Transformers: DLinear [34], LightTS [36] and S4 [12], (iv) Self-supervi”
Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (2024, positive): “Additionally, channel independence (Nie et al., 2023) is adopted to transform a multivariate input into univariate series, allowing T IME -”
Transformers in Time-Series Analysis: A Tutorial (2022, background): “Transformers have substantially improved long-term and multi-variate time-series forecasting [29], [30].”
MOMENT: A Family of Open Time-series Foundation Models (2024, methodology): “To avoid data contamination, we carefully partition all datasets into disjoint train, validation, and test splits.”
iTransformer: Inverted Transformers Are Effective for Time Series Forecasting (2023, methodology): “Channel Independence (CI) (Nie et al., 2023), regarding variates of time series independently and adopting the shared backbone, have gained increasing popularity in forecasting with performance promotions as an architecture-free method.”
Cross-Modal Epileptic Signal Harmonization: Frequency Domain Mapping Quantization for Pre-training a Unified Neurophysiological Transformer (2025, unknown): “To benchmark the performance of our proposed EpiNT model, we compared it with five representative pre-trained models designed for neurophysiological signals and general time series analysis: MAE-EEG [16], Brant [21], MOMENT [50], PatchTST [51], and VQ-MTM [19].”
Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings (2026, unknown): “For hybrid methods, we consider MOMENT (Goswami et al., 2024) and PatchTST (Nie et al., 2023b).”