Statistical and Machine Learning forecasting methods: Concerns and ways forward
By Spyros Makridakis, Evangelos Spiliotis, Vassilios Assimakopoulos. 2018
1,376 citations recorded by the source corpus.
Machine Learning (ML) methods have been proposed in the academic literature as alternatives to statistical ones for time series forecasting. Yet, scant evidence is available about their relative performance in terms of accuracy and computational requirements. The purpose of this paper is to evaluate such performance across multiple forecasting horizons using a large subset of 1045 monthly time series used in the M3 Competition. After comparing the post-sample accuracy of popular ML methods with that of eight traditional statistical ones, we found that the former are dominated across both accuracy measures used and for all forecasting horizons examined. Moreover, we observed that their computational requirements are considerably greater than those of statistical methods. The paper discusses the results, explains why the accuracy of ML models is below that of statistical ones and proposes some possible ways forward. The empirical results found in our research stress the need for objective and unbiased ways to test the performance of forecasting methods that can be achieved through sizable and open competitions allowing meaningful comparisons and definite conclusions.
Subfields: Management Science and Operations Research, Signal Processing, Time-Series & Forecasting
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Management Science and Operations Research
- Bryan Lim
- Bernie Wang
- Stefan Zohren
- Tim Januschowski
- Jan Gasthaus
- Syama Sundar Rangapuram
- Laurent Callot
- Danielle C. Maddix
In Signal Processing
- Bernie Wang
- Bryan Lim
- Tim Januschowski
- Jan Gasthaus
- Stefan Zohren
- Syama Sundar Rangapuram
- Danielle C. Maddix
- Laurent Callot
In Time-Series & Forecasting
- Bryan Lim
- Bernie Wang
- Jan Gasthaus
- Tim Januschowski
- Stefan Zohren
- Syama Sundar Rangapuram
- David Salinas
- Danielle C. Maddix
Citations in the corpus
- Time-series forecasting with deep learning: a survey (2020, background): “Secondly, similar to stationarity requirements of statistical models, machine learning models can be sensitive to how inputs are pre-processed [17, 28, 52], which ensure that data distributions at training and test time are similar.”
- N-BEATS: Neural basis expansion analysis for interpretable time series forecasting (2019, background): “And yet, unlike areas such as computer vision or natural language processing where deep learning (DL) techniques are now well entrenched, there still exists evidence that such techniques struggle to outperform classical statistical approaches for TS forecasting [22, 23].”
- Random Forest Spatial Interpolation (2020, methodology): “In the real-world case studies, the reality was not simulated from a geostatistical model as in the synthetic case, which means that STRK does not have to be the best interpolation method [53].”
- Probabilistic Forecasting with Temporal Convolutional Neural Network (2019, methodology): “ing competition (Suilin, 2017). A hybrid model that combines exponential smoothing method and RNN won the M4 forecasting competition, which consists of 100,000 series with dierent seasonal patterns (Makridakis et al., 2018a). However, training with back propagation through time (BPTT) algorithm ofte”
- A Review of ARIMA vs. Machine Learning Approaches for Time Series Forecasting in Data Driven Networks (2023, neutral): “…literature in the field of machine learning forecasting techniques is the fact that the majority of publications point to adequate accuracy values, without a proper foundation, meaning without a preestablished comparison with the results of simple statistical methods and prediction models [1].”
- Deep Learning for Time Series Forecasting: Tutorial and Literature Survey (2020, background): “1 With the overwhelming success of deep forecasting methods in the M4 competition [169], this has convinced also formerly skeptical academics [128, 129].”
- Applications of deep learning in stock market prediction: recent progress (2020, result): “…the M4 open forecasting competition with 100,000 time series which started on Jan 1, 2018 and ended on May 31, 2018, statistical approaches outperform pure ML methods (Makridakis et al., 2018a) and there are similar results with the dataset of the earlier M3 competition (Makridakis et al., 2018b).”