Unsupervised Scalable Representation Learning for Multivariate Time Series

By Jean-Yves Franceschi, Aymeric Dieuleveut, Martin Jaggi. 2019

638 citations recorded by the source corpus.

Time series constitute a challenging data type for machine learning algorithms, due to their highly variable lengths and sparse labeling in practice. In this paper, we tackle this challenge by proposing an unsupervised method to learn universal embeddings of time series. Unlike previous works, it is scalable with respect to their length and we demonstrate the quality, transferability and practicability of the learned representations with thorough experiments and comparisons. To this end, we combine an encoder based on causal dilated convolutions with a novel triplet loss employing time-based negative sampling, obtaining general-purpose representations for variable length and multivariate time series.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

On the Relationship between Self-Attention and Convolutional Layers (2019, background): “Convolutional layers acting on sequences are commonly used in the literature for text (Kim, 2014), as well as audio (van den Oord et al., 2016) and time series (Franceschi et al., 2019).”
Timeseries Anomaly Detection using Temporal Hierarchical One-Class Network (2020, methodology): “We also experiment with a recent general-purpose unsupervised timeseries representation proposed in [5].”
A Transformer-based Framework for Multivariate Time Series Representation Learning (2020, methodology): “This idea is explored further by Franceschi et al. (2019), who combine the triplet loss with a deep causal CNN with dilation, in order to make the method effective for very long time series.”
ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels (2019, methodology): “Franceschi et al. (2019) present a method for unsupervised learning of convolutional kernels for a feature transform for time series input, based on a multilayer convolutional architecture with dilation increasing exponentially in each successive layer.”
One Fits All: Power General Time Series Analysis by Pretrained LM (2023, methodology): “XGBoost Chen & Guestrin (2016), Rocket Dempster et al. (2020), LSTNet Lai et al. (2018), LSSL Gu et al. (2021), Pyraformer Liu et al. (2021), TCN Franceschi et al. (2019) and Flowformer Huang et al. (2022) are used for classification.”
Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding (2021, methodology): “Franceschi et al. (2019) employs time-based negative sampling and a triplet loss to learn scalable representations for multivariate time series.”
MOMENT: A Family of Open Time-series Foundation Models (2024, methodology): “In the zero-shot setting, MOMENT consistently outperformed all statistical interpolation methods with the exception of linear interpolation.”
ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis (2024, background): “For example, TCN and its variants (Bai et al., 2018; Sen et al., 2019; Franceschi et al., 2019) adopt causal convolution to model the DWConv and PWConv are short for depth-wise and point-wise convolution (2017).”
Self-Supervised Representation Learning: Introduction, advances, and challenges (2021, positive): “In terms of representation learning on more general time-series data, masked prediction methods based on a trans-former architecture have been shown to match supervised state of the art in a suite of benchmarks in diverse application areas [86].”
CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting (2022, methodology): “Data augmentations used are described in Appendix C. Triplet (Franceschi et al., 2020) Triplet proposes a time series self-supervised learning approach by taking positive samples to be substrings of the anchor, and negative samples to be randomly sampled from the dataset.”
TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis (2022, methodology): “In recent years, many deep models have been proposed for temporal modeling, such as MLP, TCN, RNN-based models (Hochreiter & Schmidhuber, 1997; Lai et al., 2018; Franceschi et al., 2019).”
TS2Vec: Towards Universal Representation of Time Series (2021, methodology): “Normalization Following Franceschi, Dieuleveut, and Jaggi (2019); Zhou et al. (2021), for univariate time series, we normalize datasets using z-score so that the set of observations for each dataset has zero mean and unit variance.”
A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (2022, neutral): “There are many non-Transformer-based models proposed in recent years to learn representations in time series (Franceschi et al., 2019; Tonekaboni et al., 2021; Yang & Hong, 2022; Yue et al., 2022).”
Deep Learning for Time Series Forecasting: Tutorial and Literature Survey (2020, methodology): “Convolutional architectures are able to learn relevant features from the raw time series data [11, 62, 183], but are ultimately limited to local receptive fields and can only capture long-range dependencies via many stacks of convolutional layers.”
Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings (2026, unknown): “Frozen + off-the-shelf non-linear classifier (SVM) Similar to (Goswami et al., 2024), we flatten our embeddings, Z and feed them to an SVM with the standard set of hyperparameters proposed in (Franceschi et al., 2019), which are also used in T-Rep , TS2Vec , T-Loss , etc.”
Time to Embed: Unlocking Foundation Models for Time Series with Channel Descriptions (2025, unknown): “Analogous approaches have been adapted for time series, predominantly using contrastive self-supervised tasks (Yue et al., 2022; Fraikin et al., 2024; Franceschi et al., 2019; Tonekaboni et al., 2021).”