A Transformer-based Framework for Multivariate Time Series Representation Learning

By George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, Carsten Eickhoff. Knowledge Discovery and Data Mining, 2020

1,454 citations recorded by the source corpus.

We present a novel framework for multivariate time series representation learning based on the transformer encoder architecture. The framework includes an unsupervised pre-training scheme, which can offer substantial performance benefits over fully supervised learning on downstream tasks, both with but even without leveraging additional unlabeled data, i.e., by reusing the existing data samples. Evaluating our framework on several public multivariate time series datasets from various domains and with diverse characteristics, we demonstrate that it performs significantly better than the best currently available methods for regression and classification, even for datasets which consist of only a few hundred training samples. Given the pronounced interest in unsupervised learning for nearly all domains in the sciences and in industry, these findings represent an important landmark, presenting the first unsupervised method shown to push the limits of state-of-the-art performance for multivariate time series regression and classification.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/2051548f7681c96d603de932ee23406c525276f9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

MOMENT: A Family of Open Time-series Foundation Models (2024, methodology): “Two interesting directions include using augmentation and synthetic data (Ansari et al., 2024) to improve the quality of pre-training, and looking at tuning dataset mixtures in the Time Series Pile.”
Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (2024, positive): “Self-supervised learning has been extensively developed for time series (Zhang et al., 2024), employing masked reconstruction (Zerveas et al., 2021; Nie et al., 2023) or contrastive learning (Zhang et al., 2022; Wang et al., 2023b; Yue et al., 2022).”
A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection (2023, background): “shown promise in its applicability to time series classification problems, especially for datasets with substantial training labels [185], [186].”
Unified Training of Universal Time Series Forecasting Transformers (2024, methodology): “Denoising autoencoders (Zerveas et al., 2021) and contrastive learning (Yue et al., 2022; Woo et al., 2022) have been shown to be effective pretext tasks for time series forecasting, but have largely been applied to the existing paradigm of pre-training and fine-tuning on the same dataset, without…”
Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting (2022, background): “However, masked values in time series can often be trivially predicted by simple interpolation [39], making the pre-training model only focuses on low-level information.”
Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency (2022, methodology): “Our datasets contain long time series (samples in FD-A and FD-B have 5,120 observations), and preliminary experiments identified ResNet as a better option than a Transformer variant [68].”
A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (2022, methodology): “However, although people have made attempts on Transformer-based models like time series Transformer (TST) (Zerveas et al., 2021) and TS-TCC (Eldele et al., 2021), the potential is still not fully realized yet.”
ETSformer: Exponential Smoothing Transformers for Time-series Forecasting (2022, methodology): “sequential data, Transformer-based architectures (Li et al., 2019; Wu et al., 2020; Zhou et al., 2021; Wu et al., 2021; Zerveas et al., 2021) have been actively explored for the time-series forecasting, especially for the more challenging Long Sequence Time-series Forecasting (LSTF) task.”
CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting (2022, methodology): “TST (Zerveas et al., 2021) TST is a Transformer based approach using a reconstruction loss.”
TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis (2022, methodology): “Then, we pre-process the datasets following the descriptions in (Zerveas et al., 2021), where different subsets have different sequence lengths.”
Transformers in Time Series: A Survey (2022, methodology): “[Zerveas et al. , 2021] introduces an embedding layer in Transformer that learns embedding vectors for each position index jointly with other model parameters.”