Large Language Models Are Zero-Shot Time Series Forecasters

By Nate Gruver, Marc Finzi, Shikai Qiu, Andrew Gordon Wilson. Neural Information Processing Systems, 2023

788 citations recorded by the source corpus.

By encoding time series as a string of numerical digits, we can frame time series forecasting as next-token prediction in text. Developing this approach, we find that large language models (LLMs) such as GPT-3 and LLaMA-2 can surprisingly zero-shot extrapolate time series at a level comparable to or exceeding the performance of purpose-built time series models trained on the downstream tasks. To facilitate this performance, we propose procedures for effectively tokenizing time series data and converting discrete distributions over tokens into highly flexible densities over continuous values. We argue the success of LLMs for time series stems from their ability to naturally represent multimodal distributions, in conjunction with biases for simplicity, and repetition, which align with the salient features in many time series, such as repeated seasonal trends. We also show how LLMs can naturally handle missing data without imputation through non-numerical text, accommodate textual side information, and answer questions to help explain predictions. While we find that increasing model size generally improves performance on time series, we show GPT-4 can perform worse than GPT-3 because of how it tokenizes numbers, and poor uncertainty calibration, which is likely the result of alignment interventions such as RLHF.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/123acfbccca0460171b6b06a4012dbb991cde55b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “Significant advancements have been made in M-LLMs with temporal analysis capabilities, including models such as Time-LLM [90], LLM4TS [91], TEMPO [92], and PromptCast [93], among others [94,95].”
Foundation Models for Time Series Analysis: A Tutorial and Survey (2024, neutral): “Transformer-based Pre-trainedLLM,AM,VLM General : Time-LLM [46], OFA [124], LLM4TS [11], PromptCast [105], TEMPO [10], LLMTime [35], Voice2Series [108], AutoTimes [64], UniTime [61] Finance : Yu et al. [112], Chen et al. [19], Xie et al. [103], Wimmer et al. [99] Healthcare : Liu et al. [63]…”
MOMENT: A Family of Open Time-series Foundation Models (2024, methodology): “Subsequently, Shen et al. (2023) introduced ORCA, a general cross-modal fine-tuning framework that extends the applicability of a single large-scale pretrained model to diverse modalities by adapting to a target task via an align-then-refine workflow.”
Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (2023, background): “…should explore optimal reprogramming representations, enrich LLMs with explicit time series knowledge through continued pre-training, build towards multi-modal models with joint reasoning across modalities, and apply the reprogramming framework to imbue LLMs with additional new capabilities.”
Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, methodology): “In addition, time-series LLM is also a promising technique for telecom, enabling one-model-for-all prediction [37].”
Unified Training of Universal Time Series Forecasting Transformers (2024, methodology): “As of writing, only 3 other models considered probabilistic forecasting – Lag-llama, TimeGPT, and LLMTime.”
A decoder-only foundation model for time-series forecasting (2023, methodology): “Our work also suggests that unlike recent work (Gruver et al., 2023) that recommends Large Language Models such as GPT-3 and LLama-2 as out-of-the-box zero-shot forecasters, foundation models trained from scratch exclusively on time-series data can obtain much better zero-shot performance at a…”