Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency
By Ziyuan Zhao, Theodoros Tsiligkaridis, Marinka Zitnik. Neural Information Processing Systems, 2022
507 citations recorded by the source corpus.
Pre-training on time series poses a unique challenge due to the potential mismatch between pre-training and target domains, such as shifts in temporal dynamics, fast-evolving trends, and long-range and short-cyclic effects, which can lead to poor downstream performance. While domain adaptation methods can mitigate these shifts, most methods need examples directly from the target domain, making them suboptimal for pre-training. To address this challenge, methods need to accommodate target domains with different temporal dynamics and be capable of doing so without seeing any target examples during pre-training. Relative to other modalities, in time series, we expect that time-based and frequency-based representations of the same example are located close together in the time-frequency space. To this end, we posit that time-frequency consistency (TF-C) -- embedding a time-based neighborhood of an example close to its frequency-based neighborhood -- is desirable for pre-training. Motivated by TF-C, we define a decomposable pre-training model, where the self-supervised signal is provided by the distance between time and frequency components, each individually trained by contrastive estimation. We evaluate the new method on eight datasets, including electrodiagnostic testing, human activity recognition, mechanical fault detection, and physical status monitoring. Experiments against eight state-of-the-art methods show that TF-C outperforms baselines by 15.4% (F1 score) on average in one-to-one settings (e.g., fine-tuning an EEG-pretrained model on EMG data) and by 8.4% (precision) in challenging one-to-many settings (e.g., fine-tuning an EEG-pretrained model for either hand-gesture recognition or mechanical fault prediction), reflecting the breadth of scenarios that arise in real-world applications. Code and datasets: https://github.com/mims-harvard/TFC-pretraining.
Subfields: Biomedical Engineering, Cardiology and Cardiovascular Medicine, Cognitive Neuroscience
View Paper: https://www.semanticscholar.org/paper/648d90b713997a771e2c49f02cd771e8b7b10b37
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Biomedical Engineering
Citations in the corpus
- Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (2024, unknown): “Self-supervised learning has been extensively developed for time series (Zhang et al., 2024), employing masked reconstruction (Zerveas et al., 2021; Nie et al., 2023) or contrastive learning (Zhang et al., 2022; Wang et al., 2023b; Yue et al., 2022).”
- Foundation Models for Time Series Analysis: A Tutorial and Survey (2024, unknown): “CNN-based architectures, in particular, have garnered significant attention in self-supervised learning for general time series representation, with a notable emphasis on the usage of ResNet [27, 119] and dilated convolution layers [71, 115] as foundational backbones.”
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (2023, background): “The first step among them involves time series pre-training using different strategies like supervised (Fawaz et al., 2018) or self-supervised learning (Zhang et al., 2022b; Deldari et al., 2022; Zhang et al., 2023).”
- Deep Learning for Time Series Anomaly Detection: A Survey (2022, unknown): “Manuscript submitted to ACM”