Root mean square error (RMSE) or mean absolute error (MAE)? – Arguments against avoiding RMSE in the literature
By Tianfeng Chai, Roland R. Draxler. 2014
5,740 citations recorded by the source corpus.
Abstract. Both the root mean square error (RMSE) and the mean absolute error (MAE) are regularly employed in model evaluation studies. Willmott and Matsuura (2005) have suggested that the RMSE is not a good indicator of average model performance and might be a misleading indicator of average error, and thus the MAE would be a better metric for that purpose. While some concerns over using RMSE raised by Willmott and Matsuura (2005) and Willmott et al. (2009) are valid, the proposed avoidance of RMSE in favor of MAE is not the solution. Citing the aforementioned papers, many researchers chose MAE over RMSE to present their model evaluation statistics when presenting or adding the RMSE measures could be more beneficial. In this technical note, we demonstrate that the RMSE is not ambiguous in its meaning, contrary to what was claimed by Willmott et al. (2009). The RMSE is more appropriate to represent model performance than the MAE when the error distribution is expected to be Gaussian. In addition, we show that the RMSE satisfies the triangle inequality requirement for a distance metric, whereas Willmott et al. (2009) indicated that the sums-of-squares-based statistics do not satisfy this rule. In the end, we discussed some circumstances where using the RMSE will be more beneficial. However, we do not contend that the RMSE is superior over the MAE. Instead, a combination of metrics, including but certainly not limited to RMSEs and MAEs, are often required to assess model performance.
Subfields: Global and Planetary Change, Statistics and Probability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Global and Planetary Change
- Binh Thai Pham
- Ataollah Shirzadi
- Himan Shahabi
- Baharin bin Ahmad
- D. Tien Bui
- Zaher Mundher Yaseen
- N. Al‐Ansari
- Mohammad Ali Ghorbani
In Statistics and Probability
Citations in the corpus
- The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation (2021, positive): “…than MAE; in addition to this general note, several further considerations helping researchers in choosing the more suitable metric for evaluating a regression model given the available data and the target task can be drawn ( Chai & Draxler, 2014 ; Willmott & Matsuura, 2005 ; Wang & Lu, 2018 ).”
- Implementation of a hybrid MLP-FFA model for water level prediction of Lake Egirdir, Turkey (2018, methodology): “Mean absolute error (MAE) expressed as (Chai and Draxler 2014): MAE ¼ 1 N XN i¼1 Pi Qið Þj j ð9Þ MAE represents the average of the absolute error that indicates the discrepancy between the observed and predicted data in the testing phase.”
- Revisiting Over-Smoothness in Text to Speech (2022, methodology): “distribution and MSE from the Gaussian distribution (Chai and Draxler, 2014), which means minimizing MAE/MSE will maximize the data loglikelihood under a Laplace/Gaussian distribution.”
- Design and evaluation of SVR, MARS and M5Tree models for 1, 2 and 3-day lead time forecasting of river flow data in a semiarid mountainous catchment (2018, methodology): “In accordance with literature (Chai and Draxler 2014), the RMSE value is more appropriate when the error distribution is found to be Gaussian whereas the MAE value (due its squaring effect) should be used to assess the model errors that are not normally distributed.”
- From ERA-Interim to ERA5: considerable impact of ECMWF's next-generation reanalysis on Lagrangian transport simulations (2018, neutral): “For a more detailed discussion on the advantages and disadvantages of using MADs versus standard deviations see Willmott and Matsuura (2005) and Chai and Draxler (2014).”
- A Hybrid Intelligence Approach to Enhance the Prediction Accuracy of Local Scour Depth at Complex Bridge Piers (2020, background): “Compared with RMSE, MAE can be given to be a more natural and unambiguous index to measure errors between estimated and actual observed values [78,79].”
- A survey on missing data in machine learning (2021, background): “However, [144] partially disputed the conclusions and introduced arguments against avoiding RMSE.”