A new accuracy measure based on bounded relative error for time series forecasting
By Chao Chen, Jamie Paul Twycross, J. Garibaldi. 2017
243 citations recorded by the source corpus.
Many accuracy measures have been proposed in the past for time series forecasting comparisons. However, many of these measures suffer from one or more issues such as poor resistance to outliers and scale dependence. In this paper, while summarising commonly used accuracy measures, a special review is made on the symmetric mean absolute percentage error. Moreover, a new accuracy measure called the Unscaled Mean Bounded Relative Absolute Error (UMBRAE), which combines the best features of various alternative measures, is proposed to address the common issues of existing measures. A comparative evaluation on the proposed and related measures has been made with both synthetic and real-world data. The results indicate that the proposed measure, with user selectable benchmark, performs as well as or better than other measures on selected criteria. Though it has been commonly accepted that there is no single best accuracy measure, we suggest that UMBRAE could be a good choice to evaluate forecasting methods, especially for cases where measures based on geometric mean of relative errors, such as the geometric mean relative absolute error, are preferred.
Subfields: Electrical and Electronic Engineering, Management Science and Operations Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Electrical and Electronic Engineering
In Management Science and Operations Research
- Hansika Hewamalage
- Kasun Bandara
- Christoph Bergmeir
- Alexei Botchkarev
- Muhammad Mohsin Waqas
- Usa Wannasingha Humphries
Citations in the corpus
- The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation (2021, unknown): “Despite the yet not reached agreement on its optimal mathematical expression ( Makridakis & Hibon, 2000 ; Hyndman & Koehler, 2006 ; Hyndman, 2014 ; Chen, Twycross & Garibaldi, 2017 ), SMAPE is progressively gaining momentum in the machine learning community due to its interesting properties (…”
- Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions (2019, background): “However, as Chen et al. (2017) state in their work, due to the usage of the geometric mean, values from the GMRAE can be quite small close to zero, especially in situations like ours where the expected forecasting horizon is very long such as in the NN5 and the Wikipedia Web Traffic datasets and the”
- Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typology (2018, methodology): “…titles introducing new measures include: “Novel metrics…” (Grigsby et al, 2018), “A better measure…” (Tofallis, 2015), “A new accuracy measure…” (Chen, Twycross and Garibaldi, 2017), “A new metric…” (Kim and Kim, 2016), “New Statistical Indices…” (Kyriakidis, et al, 2015), “New accuracy…”