iTransformer: Inverted Transformers Are Effective for Time Series Forecasting

By Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, Mingsheng Long. International Conference on Learning Representations, 2023

2,065 citations recorded by the source corpus.

The recent boom of linear forecasting models questions the ongoing passion for architectural modifications of Transformer-based forecasters. These forecasters leverage Transformers to model the global dependencies over temporal tokens of time series, with each token formed by multiple variates of the same timestamp. However, Transformers are challenged in forecasting series with larger lookback windows due to performance degradation and computation explosion. Besides, the embedding for each temporal token fuses multiple variates that represent potential delayed events and distinct physical measurements, which may fail in learning variate-centric representations and result in meaningless attention maps. In this work, we reflect on the competent duties of Transformer components and repurpose the Transformer architecture without any modification to the basic components. We propose iTransformer that simply applies the attention and feed-forward network on the inverted dimensions. Specifically, the time points of individual series are embedded into variate tokens which are utilized by the attention mechanism to capture multivariate correlations; meanwhile, the feed-forward network is applied for each variate token to learn nonlinear representations. The iTransformer model achieves state-of-the-art on challenging real-world datasets, which further empowers the Transformer family with promoted performance, generalization ability across different variates, and better utilization of arbitrary lookback windows, making it a nice alternative as the fundamental backbone of time series forecasting. Code is available at this repository: https://github.com/thuml/iTransformer.

Subfields: Artificial Intelligence, Management Science and Operations Research, Signal Processing

View Paper: https://www.semanticscholar.org/paper/afeeb8f5018eebb1a1d334b94dbbfc48d167efef

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Signal Processing

Citations in the corpus

Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (2024, positive): “…time series, and (2) multivariate models , which include both transformer-based approaches (Wen et al., 2023; Zhou et al., 2021; Nie et al., 2023; Liu et al., 2024b; Wang et al., 2024c; Chen et al., 2024; Wang et al., 2022) and non-transformer models (Sen et al., 2019; Jin et al., 2022; Wang et…”
MOMENT: A Family of Open Time-series Foundation Models (2024, background): “Some ways that prior work (Oreshkin et al., 2021) had overcome this issue was by leveraging data from frequencies with plenty of data.”
Unified Training of Universal Time Series Forecasting Transformers (2024, methodology): “We report the Mean Squared Error (MSE) and MAE, comparing against six state-of-the-art baselines, iTransformer (Liu et al., 2023b), TimesNet (Wu et al., 2023), PatchTST, Crossformer (Zhang & Yan, 2023), TiDE, DLinear (Zeng et al., 2023), SCINet (Liu et al., 2022), and FEDformer (Zhou et al., 2022b).”
Is Mamba Effective for Time Series Forecasting? (2024, neutral): “Motivated by these advancements, we seek to investigate the potential of Mamba-based models in the TSF domain, to uncover new research avenues for this field.”
Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings (2026, unknown): “Early foundational attempts predominantly addressed univariate series (Das et al., 2024; Ansari et al., 2024), though recent advancements have successfully extended to multivariate settings with sophisticated cross-channel modeling techniques (Woo et al., 2024b; Liu et al., 2024).”
TRACE: Grounding Time Series in Context for Multimodal Embedding and Retrieval (2025, unknown): “Traditional baselines include DLinear [29], iTransformer [24], PatchTST [22], TimesNet [53], TimeMixer [54], and multimodal model FSCA [37].”
Time to Embed: Unlocking Foundation Models for Time Series with Channel Descriptions (2025, unknown): “Early foundational attempts predominantly addressed univariate series (Das et al., 2024; Ansari et al., 2024), though recent advancements have successfully extended to multivariate settings with sophisticated cross-channel modeling techniques (Woo et al., 2024; Liu et al., 2024).”