Unified Training of Universal Time Series Forecasting Transformers

By Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, Doyen Sahoo. International Conference on Machine Learning, 2024

656 citations recorded by the source corpus.

Deep learning for time series forecasting has traditionally operated within a one-model-per-dataset framework, limiting its potential to leverage the game-changing impact of large pre-trained models. The concept of universal forecasting, emerging from pre-training on a vast collection of time series datasets, envisions a single Large Time Series Model capable of addressing diverse downstream forecasting tasks. However, constructing such a model poses unique challenges specific to time series data: i) cross-frequency learning, ii) accommodating an arbitrary number of variates for multivariate time series, and iii) addressing the varying distributional properties inherent in large-scale data. To address these challenges, we present novel enhancements to the conventional time series Transformer architecture, resulting in our proposed Masked Encoder-based Universal Time Series Forecasting Transformer (Moirai). Trained on our newly introduced Large-scale Open Time Series Archive (LOTSA) featuring over 27B observations across nine domains, Moirai achieves competitive or superior performance as a zero-shot forecaster when compared to full-shot models. Code, data, and model weights can be found at https://github.com/SalesforceAIResearch/uni2ts.

Subfields: Artificial Intelligence, Management Science and Operations Research, Signal Processing

View Paper: https://www.semanticscholar.org/paper/4a111f7a3b56d0468f13104999844885157ef17d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Artur Dubrawski
2. Mononito Goswami
3. Xiaoming Shi
4. Ming Jin
5. Shiyu Wang
6. Konrad Szafer
7. Yifu Cai
8. Arjun Choudhry

In Management Science and Operations Research

1. Artur Dubrawski
2. Shiyu Wang

In Signal Processing

1. Artur Dubrawski
2. Xiaoming Shi
3. Shiyu Wang

Citations in the corpus

• MOMENT: A Family of Open Time-series Foundation Models (2024, background): “It has 13 unique domains of data, which includes 20.085 GB worth of 13M unique time series and 1.23 billion timestamps (including channels).”
• Foundation Models for Time Series Analysis: A Tutorial and Survey (2024, neutral): “It is organized into four hierarchical levels, starting with the data category, followed by the model architecture, pre-training techniques, and finally, the application domain.”
• Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (2024, positive): “…then 6: ˆ H ← ˆ H + p j 7: add p j to J end for 11: end while 12: return J C P ROCESSED D ATA A RCHIVE Going beyond the previous work (Ansari et al., 2024; Woo et al., 2024; Liu et al., 2024d), we organized a comprehensive large-scale time series dataset from a vast collection of complex raw data.”
• Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings (2026, unknown): “Finally, for reconstruction-based methods, we evaluate Toto (Cohen et al., 2025), TIMEMIXER++ (Wang et al., 2025), Moirai (Woo et al., 2024a), and VisionTS (Chen et al., 2025).”
• Time to Embed: Unlocking Foundation Models for Time Series with Channel Descriptions (2025, unknown): “Early foundational attempts predominantly addressed univariate series (Das et al., 2024; Ansari et al., 2024), though recent advancements have successfully extended to multivariate settings with sophisticated cross-channel modeling techniques (Woo et al., 2024; Liu et al., 2024).”