Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling
By Hoshin V. Gupta, H. Kling, Koray K. Yilmaz, G. Martinez. 2009
5,372 citations recorded by the source corpus.
The mean squared error (MSE) and the related normalization, the Nash-Sutcliffe efficiency (NSE), are the two criteria most widely used for calibration and evaluation of hydrological models with observed data. Here, we present a diagnostically interesting decomposition of NSE (and hence MSE), which facilitates analysis of the relative importance of its different components in the context of hydrological modelling, and show how model calibration problems can arise due to interactions among these components. The analysis is illustrated by calibrating a simple conceptual precipitation-runoff model to daily data for a number of Austrian basins having a broad range of hydro-meteorological characteristics. Evaluation of the results clearly demonstrates the problems that can be associated with any calibration based on the NSE (or MSE) criterion. While we propose and test an alternative criterion that can help to reduce model calibration problems, the primary purpose of this study is not to present an improved measure of model performance. Instead, we seek to show that there are systematic problems inherent with any optimization based on formulations related to the MSE. The analysis and results have implications to the manner in which we calibrate and evaluate environmental models; we discuss these and suggest possible ways forward that may move us towards an improved and diagnostically meaningful approach to model performance evaluation and identification.
Subfields: Global and Planetary Change, Water Science and Technology
View Paper: https://www.semanticscholar.org/paper/3bed3d2ae40f4b771a3b3da8498803031a49637e
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Global and Planetary Change
- Hoshin V. Gupta
- Jasper A. Vrugt
- M. Clark
- William H. Farmer
- Dedi Liu
- Chaopeng Shen
- Hilary K. McMillan
- C. Onyutha
In Water Science and Technology
- Hoshin V. Gupta
- Jasper A. Vrugt
- M. Clark
- William H. Farmer
- Dedi Liu
- Chaopeng Shen
- Hilary K. McMillan
- Richard M. Vogel
Citations in the corpus
- From R-squared to coefficient of model accuracy for assessing "goodness-of-fits" (2020, background): “…on the modelers' radar for decades (see, for instance, Garrick et al.(1978), Martinec and Rango (1989), Legates and McCabe (1999), Krause et al.(2005), Criss and Winston (2008), Le Moine (2008), Gupta et al. (2009), Ehret and Zehe (2011), Legates and McCabe (1999), and Legates and McCabe (2013)).”
- A rational performance criterion for hydrological model (2020, methodology): “This can be written as a regression model = + X k X k , s o 1 0 with a desirable model fit for k1 = 1 and k0 = 0 (Murphy, 1988; Gupta et al., 2009).”
- Which objective function to calibrate rainfall–runoff models for low-flow index simulations? (2017, methodology): “Then, this study focused on the Kling and Gupta efficiency, KGE (Gupta et al., 2009), since it has been analysed and recommended in many studies (Lobligeois, 2014 ; Magand, 2014 ; Osuch et al.”
- Transferability of hydrological models and ensemble averaging methods between contrasting climatic periods (2016, background): “In contrast, the Nash-Sutcliffe criterion squares the deviation thereby weighting positive and negative outliers equally, thus providing a measure of performance in reproducing patterns of variability in the observed series [Gupta et al., 2009].”
- Balance between calibration objectives in a conceptual hydrological model (2010, background): “( 2009 ) show that, actually, the Nash-Sutcliffe coefficient is already a multi-objective function by decomposing it into three distinctive components representing the correlation, bias and a measure of relative variability in the simulated and observed values.”
- Use of an entropy‐based metric in multiobjective calibration to improve model performance (2014, methodology): “The selected KGE cutoff threshold values are comparable to thresholds used in previous hydrological applications [Gupta et al., 2009; Pechlivanidis et al., 2010a].”
- Evaluating model performance: towards a non-parametric variant of the Kling-Gupta efficiency (2018, methodology): “Gupta et al. (2009) demonstrated that a high model performance for discharge dynamics is inevitably related to an underestimation of discharge variability and that the importance of discharge volume in model calibration depends on a catchment’s discharge variability.”
- Decompositions of Taylor diagram and DISO performance criteria (2021, methodology): “These include mean squared error (MSE) and Nash– Sutcliffe efficiency (NSE, defined by Nash and Sutcliffe [1970]) applied in calibration of hydrological models (Gupta et al., 2009; Dembele et al., 2020), as well as 5726 © 2021 Royal Meteorological Society Int J Climatol.”
- Predicting carbon dioxide and energy fluxes across global FLUXNET sites with regression algorithms (2016, background): “(3) Following Gupta et al. (2009) the importance of bias in the overall uncertainty was evaluated as the ratio between the square of BIAS and the mean square error, the latter estimated as the square value of RMSE.”
- Human impact parameterizations in global hydrological models improve estimates of monthly discharges and hydrological extremes: a multi-model validation study (2018, methodology): “Moreover, use of its three sub-components enables the identification of reasons for sub-optimal model performance (Gupta et al 2009, Kling et al 2012, Thiemig et al 2013).”
- Estimating epistemic and aleatory uncertainties during hydrologic modeling: An information theoretic approach (2013, methodology): “Of course, all of these metrics are interrelated—the NSE is approximately equal to 2 [see Gupta et al., 2009; Gupta and Kling, 2011], and the MI I Qsim ; Qobsð Þ can be interpreted as a generalized correlation coefficient (that provides a more accurate assessment of relationship strength when the…”