Transformers in Time Series: A Survey

By Qingsong Wen, Tian Zhou, Chao Zhang, Weiqiu Chen, Ziqing Ma, Junchi Yan, Liang Sun. International Joint Conference on Artificial Intelligence, 2022

1,549 citations recorded by the source corpus.

Transformers have achieved superior performances in many tasks in natural language processing and computer vision, which also triggered great interest in the time series community. Among multiple advantages of Transformers, the ability to capture long-range dependencies and interactions is especially attractive for time series modeling, leading to exciting progress in various time series applications. In this paper, we systematically review Transformer schemes for time series modeling by highlighting their strengths as well as limitations. In particular, we examine the development of time series Transformers in two perspectives. From the perspective of network structure, we summarize the adaptations and modifications that have been made to Transformers in order to accommodate the challenges in time series analysis. From the perspective of applications, we categorize time series Transformers based on common tasks including forecasting, anomaly detection, and classification. Empirically, we perform robust analysis, model size analysis, and seasonal-trend decomposition analysis to study how Transformers perform in time series. Finally, we discuss and suggest future directions to provide useful research guidance.

Subfields: Artificial Intelligence, Economics and Econometrics, Signal Processing

View Paper: https://www.semanticscholar.org/paper/ed4087f6e8d77452810979f58246c5b2ad846cf8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Donghao Luo
2. Xue Wang
3. Shiyu Wang
4. Ming Jin
5. Liang Sun
6. Tian Zhou
7. Ziqing Ma
8. Artur Dubrawski

In Economics and Econometrics

1. Geoffrey I. Webb
2. Cesare Alippi
3. Donghao Luo
4. Xue Wang

In Signal Processing

1. Donghao Luo
2. Xue Wang
3. Shiyu Wang
4. Tian Zhou
5. Liang Sun
6. Ziqing Ma
7. Artur Dubrawski
8. Nam H. Nguyen

Citations in the corpus

• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (2022, background): “…processing (NLP) (Kalyan et al., 2021), computer vision (CV) (Khan et al., 2021), speech (Karita et al., 2019), and more recently time series (Wen et al., 2022), benefiting from its attention mechanism which can automatically learn the connections between elements in a sequence, thus becomes…”
• ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis (2024, methodology): “Among them, the rise of Transformer-based methods and MLP-based models is especially compelling (Nie et al., 2023; Zhang & Yan, 2023; Zhou et al., 2022; Cirstea et al., 2022; Wu et al., 2021; Liu et al., 2021a; Li et al., 2019b; Kitaev et al., 2020; Vaswani et al., 2017) (Li et al., 2023b; Zhang et…”
• One Fits All: Power General Time Series Analysis by Pretrained LM (2023, background): “…(2017); Devlin et al. (2019); Dosovitskiy et al. (2021); Rao et al. (2021), transformer has been introduced to various time series tasks with promising results Wen et al. (2023), especially for time series forecasting Lim et al. (2021); Zhou et al. (2022, 2021); Wu et al. (2021); Nie et al. (2022).”
• FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting (2022, background): “, 2017) has been introduced to capture long-term dependencies in time series forecasting and shows promising results (Wen et al., 2022; Zhou et al., 2021; Wu et al., 2021).”
• MOMENT: A Family of Open Time-series Foundation Models (2024, methodology): “In cases where such splits are unavailable, we randomly sample 60% of the data for training, 10% for validation, and 30% for testing.”
• A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection (2023, background): “The approaches in the first category encompass recurrence-based (e.g., RNNs [26]), convolution-based (e.g., TCNs [102]), attention-based (e.g., transformers [27]), and a combination of these (i.e., hybrid ).”
• MICN: Multi-scale Local and Global Context Modeling for Long-term Series Forecasting (2023, background): “It has also recently been applied in long-term series forecasting tasks (Li et al., 2019b) (Wen et al., 2022) and can model the long-term dependence of sequences effectively, allowing leaps and bounds in the accuracy and length of time series forecasts (Zhu & Soricut, 2021) (Wu et al., 2021b) (Zhou…”
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (2023, background): “, 2018) and transformers (Wen et al., 2023) are developed for handling longer temporal dependencies.”
• FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting (2022, background): “3 Data Normalization As Wu et al. (2021); Zhou et al.”