Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting
By Bryan Lim, Sercan Ö. Arik, Nicolas Loeff, Tomas Pfister. 2019
2,901 citations recorded by the source corpus.
Multi-horizon forecasting problems often contain a complex mix of inputs -- including static (i.e. time-invariant) covariates, known future inputs, and other exogenous time series that are only observed historically -- without any prior information on how they interact with the target. While several deep learning models have been proposed for multi-step prediction, they typically comprise black-box models which do not account for the full range of inputs present in common scenarios. In this paper, we introduce the Temporal Fusion Transformer (TFT) -- a novel attention-based architecture which combines high-performance multi-horizon forecasting with interpretable insights into temporal dynamics. To learn temporal relationships at different scales, the TFT utilizes recurrent layers for local processing and interpretable self-attention layers for learning long-term dependencies. The TFT also uses specialized components for the judicious selection of relevant features and a series of gating layers to suppress unnecessary components, enabling high performance in a wide range of regimes. On a variety of real-world datasets, we demonstrate significant performance improvements over existing benchmarks, and showcase three practical interpretability use-cases of TFT.
Subfields: Electrical and Electronic Engineering, Management Science and Operations Research, Model Architectures, Signal Processing, Time-Series & Forecasting
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Electrical and Electronic Engineering
In Management Science and Operations Research
- Bryan Lim
- Bernie Wang
- Stefan Zohren
- Tim Januschowski
- Jan Gasthaus
- Junchi Yan
- Syama Sundar Rangapuram
- Laurent Callot
In Model Architectures
- Bryan Lim
- Bernie Wang
- Laurent Callot
- Frédérick Gusset
- Ziqing Ma
- Michael Bohlke-Schneider
- David Salinas
- Xiao Ma
In Signal Processing
- Bernie Wang
- Bryan Lim
- Tim Januschowski
- Jan Gasthaus
- Junchi Yan
- Stefan Zohren
- Syama Sundar Rangapuram
- Danielle C. Maddix
In Time-Series & Forecasting
- Bryan Lim
- Bernie Wang
- Stefan Zohren
- Tim Januschowski
- Jan Gasthaus
- Danielle C. Maddix
- Syama Sundar Rangapuram
- Junchi Yan
Citations in the corpus
- Time-series forecasting with deep learning: a survey (2020, background): “By analysing distributions of attention vectors across time, [54] also shows how attention mechanisms can be used to identify persistent temporal relationships—such as seasonal patterns—in the dataset.”
- One Fits All: Power General Time Series Analysis by Pretrained LM (2023, background): “…(2017); Devlin et al. (2019); Dosovitskiy et al. (2021); Rao et al. (2021), transformer has been introduced to various time series tasks with promising results Wen et al. (2023), especially for time series forecasting Lim et al. (2021); Zhou et al. (2022, 2021); Wu et al. (2021); Nie et al. (2022).”
- A Transformer-based Framework for Multivariate Time Series Representation Learning (2020, methodology): “[20] used a transformer for multi-horizon univariate forecasting, supporting interpretation of temporal dynamics.”
- Deep Learning for Time Series Forecasting: Tutorial and Literature Survey (2020, methodology): “For example, Lim et al. [120] introduce the Temporal Fusion Transformer (TFT), which incorporates novel model components for embedding static covariates, performing “variable selection”, and gating components that skip over irrelevant parts of the context.”
- Inter-Series Attention Model for COVID-19 Forecasting (2020, methodology): “The model degenerates to an autoregressive model similar to ConvTrans and TFT; • ACTS-f We remove the additional features in the model and only rely on incidence data.”
- TSMixer: An all-MLP Architecture for Time Series Forecasting (2023, methodology): “Additionally, we include TFT (Lim et al., 2021), a deep learning-based model that considers auxiliary information, as a baseline to understand the limitations of solely relying on historical features.”
- Transformers in Time-Series Analysis: A Tutorial (2022, methodology): “Lastly we will discuss three architectures that improve upon attention, LogSparse Transformers [32], TFT [34], and YFormer [39].”
- N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting (2022, background): “…adopting the attention mechanism and the rise of Transformer-inspired approaches (Li et al. 2019; Fan et al. 2019; Alaa and van der Schaar 2019; Lim et al. 2021), as well as the introduction of attention-free architectures composed of deep stacks of fully connected layers (Oreshkin et al.…”
- Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction (2020, background): “Besides, as Transformers treat a sequence as a bag of words, they normally have problem modeling time series data where strong temporal consistency is enforced [29].”
- SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction (2021, neutral): “1(b)) are proposed in the literature [21, 23, 38, 25].”
- Darts: User-Friendly Modern Machine Learning for Time Series (2021, methodology): “…and Letham, 2018), FFT-based forecasting, RNN models similar to DeepAR (Flunkert et al., 2017), N-BEATS (Oreshkin et al., 2019), TCN (Bai et al., 2018), TFT (Lim et al., 2021) and generic regression models that can wrap around any external tabular regression model (such as scikit-learn models).”
- Traffic transformer: Capturing the continuity and periodicity of time series for traffic forecasting (2020, background): “In light of this, several attempts have been made to tailor Transformer to time series forecasting (Li et al., 2019; Li & Moura, 2019; Lim, Arik, Loeff, & Pfister, 2019).”
- Transformers in Time Series: A Survey (2022, methodology): “[Lim et al. , 2021] uses an LSTM network to encode positional embeddings, which can better exploit sequential ordering information in time series.”