TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting

By Vijayabharathi Ekambaram, Arindam Jati, Nam H. Nguyen, Phanwadee Sinthong, Jayant R. Kalagnanam. Knowledge Discovery and Data Mining, 2023

397 citations recorded by the source corpus.

Transformers have gained popularity in time series forecasting for their ability to capture long-sequence interactions. However, their memory and compute-intensive requirements pose a critical bottleneck for long-term forecasting, despite numerous advancements in compute-aware self-attention modules. To address this, we propose TSMixer, a lightweight neural architecture exclusively composed of multi-layer perceptron (MLP) modules. TSMixer is designed for multivariate forecasting and representation learning on patched time series, providing an efficient alternative to Transformers. Our model draws inspiration from the success of MLP-Mixer models in computer vision. We demonstrate the challenges involved in adapting Vision MLP-Mixer for time series and introduce empirically validated components to enhance accuracy. This includes a novel design paradigm of attaching online reconciliation heads to the MLP-Mixer backbone, for explicitly modeling the time-series properties such as hierarchy and channel-correlations. We also propose a Hybrid channel modeling approach to effectively handle noisy channel interactions and generalization across diverse datasets, a common challenge in existing patch channel-mixing methods. Additionally, a simple gated attention mechanism is introduced in the backbone to prioritize important features. By incorporating these lightweight components, we significantly enhance the learning capability of simple MLP structures, outperforming complex Transformer models with minimal computing usage. Moreover, TSMixer's modular design enables compatibility with both supervised and masked self-supervised learning methods, making it a promising building block for time-series Foundation Models. TSMixer outperforms state-of-the-art MLP and Transformer models in forecasting by a considerable margin of 8-60%. It also outperforms the latest strong benchmarks of Patch-Transformer models (by 1-2%) with a significant reduction in memory and runtime (2-3X).

Subfields: Artificial Intelligence, Management Science and Operations Research, Signal Processing

View Paper: https://www.semanticscholar.org/paper/3803d828e31381c853374133dd4022d3ccd60a5a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Haixu Wu
  2. Shiyu Wang
  3. Mingsheng Long
  4. Jianmin Wang
  5. Kate Saenko
  6. Zhangjie Cao
  7. Jian Liang
  8. Jiashi Feng

In Management Science and Operations Research

  1. Shiyu Wang
  2. Lintao Ma
  3. Haixu Wu
  4. Mingsheng Long

In Signal Processing

  1. Shiyu Wang
  2. Lintao Ma
  3. Haixu Wu
  4. Tengge Hu
  5. Mingsheng Long

Citations in the corpus