FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting

By Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, Rong Jin. International Conference on Machine Learning, 2022

3,326 citations recorded by the source corpus.

Although Transformer-based methods have significantly improved state-of-the-art results for long-term series forecasting, they are not only computationally expensive but more importantly, are unable to capture the global view of time series (e.g. overall trend). To address these problems, we propose to combine Transformer with the seasonal-trend decomposition method, in which the decomposition method captures the global profile of time series while Transformers capture more detailed structures. To further enhance the performance of Transformer for long-term prediction, we exploit the fact that most time series tend to have a sparse representation in well-known basis such as Fourier transform, and develop a frequency enhanced Transformer. Besides being more effective, the proposed method, termed as Frequency Enhanced Decomposed Transformer ({\bf FEDformer}), is more efficient than standard Transformer with a linear complexity to the sequence length. Our empirical studies with six benchmark datasets show that compared with state-of-the-art methods, FEDformer can reduce prediction error by $14.8\%$ and $22.6\%$ for multivariate and univariate time series, respectively. Code is publicly available at https://github.com/MAZiqing/FEDformer.

Subfields: Artificial Intelligence, Management Science and Operations Research, Signal Processing

View Paper: https://www.semanticscholar.org/paper/563bac1c5cdd5096e9dbf8d4f3d5b3c4f7284e06

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Signal Processing

Citations in the corpus

FourierGNN: Rethinking Multivariate Time Series Forecasting from a Pure Graph Perspective (2023, positive): “…compare FourierGNN against GNN-based models like GraphWaveNet [17], StemGNN [10], MTGNN [13], and AGCRN [2], and two representative Transformer-based models like Reformer [35] and Informer [9], as well as two frequency enhanced Transformer-based models including Autoformer [4] and FEDformer [27].”
Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting (2023, methodology): “…2015), 2) LSTnet (Lai et al., 2018), 3) MTGNN (Wu et al., 2020), and recent Transformer-based models for MTS forecasting: 4) Transformer (Vaswani et al., 2017), 5) Informer (Zhou et al., 2021), 6) Autoformer (Wu et al., 2021a), 7) Pyraformer (Liu et al., 2021a) and 8) FEDformer (Zhou et al., 2022).”
FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting (2022, methodology): “OPs (including Legendre, Laguerre, Chebyshev, etc.) and other orthogonal basis (Fourier and Multiwavelets) are widely studied in numerous ﬁelds and recently brought in deep learning Wang et al. (2018); Gupta et al. (2021); Zhou et al. (2022); Voelker et al. (2019); Gu et al. (2020).”
Unified Training of Universal Time Series Forecasting Transformers (2024, methodology): “We report the Mean Squared Error (MSE) and MAE, comparing against six state-of-the-art baselines, iTransformer (Liu et al., 2023b), TimesNet (Wu et al., 2023), PatchTST, Crossformer (Zhang & Yan, 2023), TiDE, DLinear (Zeng et al., 2023), SCINet (Liu et al., 2022), and FEDformer (Zhou et al., 2022b).”
MICN: Multi-scale Local and Global Context Modeling for Long-term Series Forecasting (2023, methodology): “Different from the MOEDecomp block of FEDformer (Zhou et al., 2022), we use simple mean operation to integrate these different patterns because we cannot determine the weight of each pattern before learning its features.”
TSMixer: An all-MLP Architecture for Time Series Forecasting (2023, methodology): “…− P ) + c , where a, c ∈ R are constants, the linear model still has a solution for perfect prediction: A more general assumption is that the time series can be decomposed into a periodic sequence and a sequence with smooth trend (Holt, 2004; Zhang & Qi, 2005; Wu et al., 2021; Zhou et al., 2022b).”
A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (2022, methodology): “Informer (Zhou et al., 2021), Autoformer (Wu et al., 2021), and FEDformer (Zhou et al., 2022) are among the best variants of the Transformer model successfully applying to time series data.”
Spatio-Temporal Adaptive Embedding Makes Vanilla Transformer SOTA for Traffic Forecasting (2023, methodology): “Researchers have devoted considerable effort to developing fancy and complicated models for traffic forecasting, such as novel graph convolutions [2, 3, 9–11, 13, 18, 21, 26, 29, 31, 37, 41], learning graph structures [15, 23, 32, 34, 35, 40], efficient attention mechanisms [5, 20, 33, 43, 44], and ”
TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis (2022, methodology): “In time series analysis, many Transformer-based models adopt the attention mechanism or its variants to capture the pair-wise temporal dependencies among time points (Li et al., 2019; Kitaev et al., 2020; Zhou et al., 2021; 2022).”
Is Mamba Effective for Time Series Forecasting? (2024, neutral): “The preceding 𝐿 and 𝑇 are referred to as the review window and prediction horizon respectively, representing the lengths of the past and future time windows, while 𝑝 is a variate and 𝑉 represents the total number of variates.”
Long-term Forecasting with TiDE: Time-series Dense Encoder (2023, methodology): “Subsequently, there has been a surge of Transformer-based forecasting papers (Wu et al., 2021; Zhou et al., 2021; 2022) in the time-series community that have claimed state-of-the-art (SoTA) forecasting performance for long-horizon tasks.”
Large Language Models Are Zero-Shot Time Series Forecasters (2023, methodology): “We baseline against numbers obtained by running public implementations from the Autoformer [49] and FEDFormer [55] codebases (Appendix C.3).”
iTransformer: Inverted Transformers Are Effective for Time Series Forecasting (2023, background): “Baselines We carefully choose 10 well-acknowledged forecasting models as our benchmark, including (1) Transformer-based methods: Autoformer (Wu et al., 2021), FEDformer (Zhou et al., 2022), Stationary (Liu et al., 2022b), Crossformer (Zhang & Yan, 2023), PatchTST (Nie et al., 2023); (2) Linear-based”
Transformers in Time Series: A Survey (2022, methodology): “, 2021] or frequency processing [Zhou et al., 2022] into time series Transformer architectures would bring superior performance improvements.”
Transformers in Time-Series Analysis: A Tutorial (2022, background): “Various modifications have been proposed in the literature to optimize Transformer performance for timeseries tasks [31], [32], [33].”