A note on the validity of cross-validation for evaluating autoregressive time series prediction

By Christoph Bergmeir, Rob J Hyndman, Bonsoo Koo. Computational Statistics & Data Analysis, 2018

589 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/22a3ade55b42a4abcdf1ae55ea3633c047191c7a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, background): “In such a setting, we are usually interested in how the model will generalize in the future, and the rolling or expanding window forecast must be used for evaluation (Bergmeir et al., 2018) instead of regular CV.”
Interpreting insect declines: seven challenges and a way forward (2020, methodology): “…of the two approaches more generally, a walk-forward cross-validation or combinatorial k-fold cross-validation procedure for time series (e.g. Bergmeir et al., 2018) could be used to determine sensitivity to outliers in the data, when time series are shorter than the 15 time points…”
Transformers in Time-Series Analysis: A Tutorial (2022, background): “Given that the length of the two sequences and the vocabulary size are not necessarily the same, the model has to learn to encode the source sequence into a fixed-length representation that it can then decode to generate the target sequence in an auto-regressive fashion [8].”
Forecasting: theory and practice (2020, methodology): “esting are omitted (Racine, 2000; Burman et al., 1994). Finally, it has been shown that with purely autoregressive models, CV can be used without modiﬁcations, i.e., with randomly choosing the folds (Bergmeir et al., 2018). Here, CV estimates the generalisation error accurately, as long as the model”
Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions (2019, unknown): “As discussed by Bergmeir et al. (2018), while being applicable to detect overfitting, a generic k-fold cross-validation may lead to an underestimation of the cross-validation errors for models that underfit.”
Evaluating time series forecasting models: an empirical study on performance estimation methods (2019, methodology): “tionary time series. Each red horizontal arrow denote a non-stationarity identied by the test. 3.2.2 Synthetic time series We use three synthetic use cases dened in previous work by Bergmeir et al. [5,6]. The data generating processes are all stationary and are designed as follows: S1: A stable auto”
Design and evaluation of SVR, MARS and M5Tree models for 1, 2 and 3-day lead time forecasting of river flow data in a semiarid mountainous catchment (2018, background): “…particular, the validity of our approach is well-established, for example, a recent study justifies that standard random sampling schemes for the purpose of pure series modelling with machine learning algorithms (i.e. model errors) of the models are found to be uncorrelated (Bergmeir et al. 2018).”