Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
By Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, Wan Zhang. AAAI Conference on Artificial Intelligence, 2020
8,143 citations recorded by the source corpus.
Many real-world applications require the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series forecasting (LSTF) demands a high prediction capacity of the model, which is the ability to capture precise long-range dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to increase the prediction capacity. However, there are several severe issues with Transformer that prevent it from being directly applicable to LSTF, including quadratic time complexity, high memory usage, and inherent limitation of the encoder-decoder architecture. To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics: (i) a ProbSparse self-attention mechanism, which achieves O(L log L) in time complexity and memory usage, and has comparable performance on sequences' dependency alignment. (ii) the self-attention distilling highlights dominating attention by halving cascading layer input, and efficiently handles extreme long input sequences. (iii) the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions. Extensive experiments on four large-scale datasets demonstrate that Informer significantly outperforms existing methods and provides a new solution to the LSTF problem.
Subfields: Electrical and Electronic Engineering, Management Science and Operations Research, Signal Processing
View Paper: https://www.semanticscholar.org/paper/5b9d8bcc46b766b47389c912a8e026f81b91b0d8
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Electrical and Electronic Engineering
In Management Science and Operations Research
In Signal Processing
Citations in the corpus
- A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (2022, methodology): “Informer (Zhou et al., 2021) proposes a ProbSparse self-attention with distilling techniques to extract the most important keys efficiently.”
- Transformers in Time-Series Analysis: A Tutorial (2022, methodology): “Many forecasting Transformers for time-series data have been developed in the recent literature [54, 11, 80, 32, 69, 34, 9, 70, 47, 62, 39].”
- Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting (2023, methodology): “…2015), 2) LSTnet (Lai et al., 2018), 3) MTGNN (Wu et al., 2020), and recent Transformer-based models for MTS forecasting: 4) Transformer (Vaswani et al., 2017), 5) Informer (Zhou et al., 2021), 6) Autoformer (Wu et al., 2021a), 7) Pyraformer (Liu et al., 2021a) and 8) FEDformer (Zhou et al., 2022).”
- Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy (2021, methodology): “For time series analysis, benefiting from the advantage of the self-attention mechanism, Transformers are used to discover the reliable long-range temporal dependencies (Kitaev et al., 2020; Li et al., 2019b; Zhou et al., 2021; Wu et al., 2021).”
- ETSformer: Exponential Smoothing Transformers for Time-series Forecasting (2022, methodology): “sequential data, Transformer-based architectures (Li et al., 2019; Wu et al., 2020; Zhou et al., 2021; Wu et al., 2021; Zerveas et al., 2021) have been actively explored for the time-series forecasting, especially for the more challenging Long Sequence Time-series Forecasting (LSTF) task.”
- TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting (2023, methodology): “We categorize SOTA forecasting benchmarks into the following categories: (i) Standard Transformers: FEDformer [38], Autoformer[30] and Informer [37], (ii) Patch Transformers: PatchTST [22] and CrossFormer [3], (iii) MLPs and Non-Transformers: DLinear [34], LightTS [36] and S4 [12], (iv) Self-supervi”
- SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction (2021, methodology): “Besides, compared with the vanilla Transformer-based methods [18, 21, 42], the newly-proposed Transformer-based forecasting model Autoformer [38] achieves the second best performance in all experimental settings and also surpasses”
- N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting (2022, methodology): “We follow the experimental settings from (Wu et al. 2021; Zhou et al. 2020) (NeurIPS 2021 and AAAI 2021 Best Paper Award).”
- FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting (2022, methodology): “…and memory requirement make it difficult for Transformer to be applied to long sequence modeling, numerous studies are devoted to reduce the computational cost of Transformer (Li et al., 2019; Kitaev et al., 2020; Zhou et al., 2021; Wang et al., 2020; Xiong et al., 2021; Ma et al., 2021).”
- A decoder-only foundation model for time-series forecasting (2023, methodology): “Subsequently, we perform ablation studies that justify different choices made in our architecture.”
- Long-term Forecasting with TiDE: Time-series Dense Encoder (2023, methodology): “Subsequently, there has been a surge of Transformer-based forecasting papers (Wu et al., 2021; Zhou et al., 2021; 2022) in the time-series community that have claimed state-of-the-art (SoTA) forecasting performance for long-horizon tasks.”
- FourierGNN: Rethinking Multivariate Time Series Forecasting from a Pure Graph Perspective (2023, neutral): “We compare our model FourierGNN with five models (including StemGNN [10], AGCRN [2], GraphWaveNet [17], MTGNN [13], and Informer [9]) on the Wiki dataset under different prediction lengths, and the results are shown in Table 9.”
- CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting (2022, methodology): “We include supervised forecasting approaches - two Transformer based models, Informer (Zhou et al., 2021) and LogTrans(Li et al.”
- FITS: Modeling Time Series with 10k Parameters (2023, methodology): “Figure 3: Waveform (1 st row) and amplitude spectrum (2 nd row) of a time series segment selected from the ’OT’ channel of the ETTh1 dataset, spanning from the 1500 th to the 1980 th data point.”
- Frequency-domain MLPs are More Effective Learners in Time Series Forecasting (2023, methodology): “Besides, the robustness of these models could be easily influenced with a large amount of parameters, especially when the available training data is limited [13, 18].”
- SAITS: Self-Attention-based Imputation for Time Series (2022, background): “Electricity Transformer Temperature (ETT) This dataset [52] is collected from electricity transformers in two years from 2016/07/01 to 2018/06/26.”