Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting

By Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X. Liu, Schahram Dustdar. International Conference on Learning Representations, 2022

1,042 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/30dcc0e191a376fea0e7a46f94c53872c029efc9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Donghao Luo
2. Shiyu Wang
3. Zhijian Xu
4. Tian Zhou
5. Haixu Wu
6. Abhimanyu Das
7. Ailing Zeng
8. Si Chen

Citations in the corpus

• ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis (2024, methodology): “…methods and MLP-based models is especially compelling (Nie et al., 2023; Zhang & Yan, 2023; Zhou et al., 2022; Cirstea et al., 2022; Wu et al., 2021; Liu et al., 2021a; Li et al., 2019b; Kitaev et al., 2020; Vaswani et al., 2017) (Li et al., 2023b; Zhang et al., 2022; Zeng et al., 2022).”
• Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting (2023, methodology): “7) Pyraformer (Liu et al., 2021a) is a Transformer-based model learning multi-resolution representation of the time series by the pyramidal attention module to capture cross-time dependency for forecasting.”
• TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting (2024, methodology): “Although there exist some models with temporal multiscale designs, such as Pyraformer (Liu et al., 2021) with pyramidal attention and SCINet (Liu et al., 2022a) with a bifurcate downsampling tree, their future predictions do not make use of the information at different scales extracted from the past”
• One Fits All: Power General Time Series Analysis by Pretrained LM (2023, methodology): “XGBoost Chen & Guestrin (2016), Rocket Dempster et al. (2020), LSTNet Lai et al. (2018), LSSL Gu et al. (2021), Pyraformer Liu et al. (2021), TCN Franceschi et al. (2019) and Flowformer Huang et al. (2022) are used for classification.”
• Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting (2022, methodology): “Pyraformer [21] designs pyramid attention module (PAM) to capture temporal dependencies with different hierarchies.”
• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (2022, methodology): “, 2021), Pyraformer (Liu et al., 2022), LogTrans (Li et al.”
• MICN: Multi-scale Local and Global Context Modeling for Long-term Series Forecasting (2023, methodology): “Some notable models include: LogTrans (Li et al., 2019b), Informer (Zhou et al., 2021), Reformer (Kitaev et al., 2020), Autoformer Wu et al. (2021b), Pyraformer (Liu et al., 2021a), FEDformer (Zhou et al., 2022).”
• SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction (2021, methodology): “1(b)) are proposed in the literature [21, 23, 38, 25].”
• Are Transformers Effective for Time Series Forecasting? (2022, methodology): “We include six state-of-the-art Transformer-based methods: FEDformer [29], Autoformer [27], Informer [28], Pyraformer [18], LogTrans [16], and Reformer [14].”
• TSMixer: An all-MLP Architecture for Time Series Forecasting (2023, methodology): “…have been proposed due to their capacity to capture both complex temporal patterns and cross-variate dependencies (Gamboa, 2017; Li et al., 2019; Wen et al., 2017; Zhou et al., 2021; Wu et al., 2021; Lim & Zohren, 2021; Liu et al., 2022a; Zhou et al., 2022b; Liu et al., 2022b; Zhou et al., 2022a) .”
• Recipe for a General, Powerful, Scalable Graph Transformer (2022, background): “Pyraformer [37] is an example of a linear transformer for time series data, whereas S4 [21] is a more general signal processing approach that employs the state space model theory without the attention mechanism.”
• FITS: Modeling Time Series with 10k Parameters (2023, methodology): “Notably, large models such as TimesNet and Pyraformer require a staggering number of parameters, with 300.6M and 241.4M, respectively.”
• PromptCast: A New Prompt-Based Learning Paradigm for Time Series Forecasting (2022, methodology): “The existing forecasting methods including the state-of-the-art Transformer-based forecasting models [8], [9], [10], [11], [12], [13] can be simplified as a numerical forecasting paradigm as shown in Fig.”
• Long-term Forecasting with TiDE: Time-series Dense Encoder (2023, methodology): “We choose SOTA Transformers based models for time-series including Fedformer (Zhou et al., 2022), Autoformer (Wu et al., 2021), Informer (Zhou et al., 2021), Pyraformer (Liu et al., 2021) and LongTrans (Li et al., 2019a).”
• Spatio-Temporal Adaptive Embedding Makes Vanilla Transformer SOTA for Traffic Forecasting (2023, methodology): “Researchers have devoted considerable effort to developing fancy and complicated models for traffic forecasting, such as novel graph convolutions [2, 3, 9–11, 13, 18, 21, 26, 29, 31, 37, 41], learning graph structures [15, 23, 32, 34, 35, 40], efficient attention mechanisms [5, 20, 33, 43, 44], and ”
• Transformers in Time-Series Analysis: A Tutorial (2022, background): “Various modifications have been proposed in the literature to optimize Transformer performance for timeseries tasks [31], [32], [33].”