One Fits All: Power General Time Series Analysis by Pretrained LM

By Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, Rong Jin. Neural Information Processing Systems, 2023

1,009 citations recorded by the source corpus.

Although we have witnessed great success of pre-trained models in natural language processing (NLP) and computer vision (CV), limited progress has been made for general time series analysis. Unlike NLP and CV where a unified model can be used to perform different tasks, specially designed approach still dominates in each time series analysis task such as classification, anomaly detection, forecasting, and few-shot learning. The main challenge that blocks the development of pre-trained model for time series analysis is the lack of a large amount of data for training. In this work, we address this challenge by leveraging language or CV models, pre-trained from billions of tokens, for time series analysis. Specifically, we refrain from altering the self-attention and feedforward layers of the residual blocks in the pre-trained language or image model. This model, known as the Frozen Pretrained Transformer (FPT), is evaluated through fine-tuning on all major types of tasks involving time series. Our results demonstrate that pre-trained models on natural language or images can lead to a comparable or state-of-the-art performance in all main time series analysis tasks, as illustrated in Figure 1. We also found both theoretically and empirically that the self-attention module behaviors similarly to principle component analysis (PCA), an observation that helps explains how transformer bridges the domain gap and a crucial step towards understanding the universality of a pre-trained transformer.The code is publicly available at https://github.com/DAMO-DI-ML/One_Fits_All.

Subfields: Applied Mathematics, Management Science and Operations Research, Signal Processing

View Paper: https://www.semanticscholar.org/paper/5b7f5488c380cf5085a5dd93e993ad293b225eee

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Mathematics

In Management Science and Operations Research

In Signal Processing

Citations in the corpus

A decoder-only foundation model for time-series forecasting (2023, background): “Zhou et al. (2023) and (Chang et al., 2023) show how to fine-tune a GPT-2 (Radford et al., 2019) backbone model for time-series forecasting tasks.”
Foundation Models for Time Series Analysis: A Tutorial and Survey (2024, positive): “…model [66, 69, 72] or specific components ( e.g. , training positional embeddings and layer normalization, while keeping feedfor-ward and attention layers frozen when fine-tuning LLMs) [11, 124], to directly infer results, or integrate foundation models as part of the whole model [19, 52, 86, 106].”
Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, background): “A similar strategy can be found in [191], which freezes all attention and feed-forward layers, and only fully fine-tunes the embedding layers and applies the LNT. Incorporating LNT in the fine-tuning process, in the context of adapting pre-trained LLM models for time-series prediction, provides a…”
MOMENT: A Family of Open Time-series Foundation Models (2024, methodology): “In the zero-shot setting, MOMENT consistently outperformed all statistical interpolation methods with the exception of linear interpolation.”
Large Language Models Are Zero-Shot Time Series Forecasters (2023, methodology): “For example, Zhou et al. [56] propose FPT, which ﬁnetunes a BERT encoder to perform time series forecasting.”
TRACE: Grounding Time Series in Context for Multimodal Embedding and Retrieval (2025, unknown): “Recently, several multimodal encoders have been proposed to integrate time series and text [35, 36, 37, 38, 39, 40], which aim to leverage the generalization capabilities of large language models by reprogramming time series into token-like representations or textual prototypes.”