Evaluating time series forecasting models: an empirical study on performance estimation methods
By Vítor Cerqueira, Luís Torgo, Igor Mozetic. Machine-mediated learning, 2019
412 citations recorded by the source corpus.
Performance estimation aims at estimating the loss that a predictive model will incur on unseen data. This process is a fundamental stage in any machine learning project. In this paper we study the application of these methods to time series forecasting tasks. For independent and identically distributed data the most common approach is cross-validation. However, the dependency among observations in time series raises some caveats about the most appropriate way to estimate performance in this type of data. Currently, there is no consensual approach. We contribute to the literature by presenting an extensive empirical study which compares different performance estimation methods for time series forecasting tasks. These methods include variants of cross-validation, out-of-sample (holdout), and prequential approaches. Two case studies are analysed: One with 174 real-world time series and another with three synthetic time series. Results show noticeable differences in the performance estimation methods in the two scenarios. In particular, empirical experiments suggest that blocked cross-validation can be applied to stationary time series. However, when the time series are non-stationary, the most accurate estimates are produced by out-of-sample methods, particularly the holdout approach repeated in multiple testing periods.
Subfields: Artificial Intelligence, Management Science and Operations Research, Signal Processing
View Paper: https://www.semanticscholar.org/paper/1ef6b9c734ceb775bc6055af93918b5db5bf190d
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Management Science and Operations Research
Citations in the corpus
- Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “Similarly, cross-validation or testing on time-series must be handled with care so as to not include future data in the training set [23].”