Why hydrological predictions should be evaluated using information theory

By Steven V. Weijs, G. Schoups, N. V. D. Giesen. 2010

127 citations recorded by the source corpus.

Abstract. Probabilistic predictions are becoming increasingly popular in hydrology. Equally important are methods to test such predictions, given the topical debate on uncertainty analysis in hydrology. Also in the special case of hydrological forecasting, there is still discussion about which scores to use for their evaluation. In this paper, we propose to use information theory as the central framework to evaluate predictions. From this perspective, we hope to shed some light on what verification scores measure and should measure. We start from the ''divergence score'', a relative entropy measure that was recently found to be an appropriate measure for forecast quality. An interpretation of a decomposition of this measure provides insight in additive relations between climatological uncertainty, correct information, wrong information and remaining uncertainty. When the score is applied to deterministic forecasts, it follows that these increase uncertainty to infinity. In practice, however, deterministic forecasts tend to be judged far more mildly and are widely used. We resolve this paradoxical result by proposing that deterministic forecasts either are implicitly probabilistic or are implicitly evaluated with an underlying decision problem or utility in mind. We further propose that calibration of models representing a hydrological system should be the based on information-theoretical scores, because this allows extracting all information from the observations and avoids learning from information that is not there. Calibration based on maximizing utility for society trains an implicit decision model rather than the forecasting system itself. This inevitably results in a loss or distortion of information in the data and more risk of overfitting, possibly leading to less valuable and informative forecasts. We also show this in an example. The final conclusion is that models should preferably be explicitly probabilistic and calibrated to maximize the information they provide.

Subfields: Atmospheric Science, Environmental Engineering, Water Science and Technology

View Paper: https://www.semanticscholar.org/paper/6a20b5029bbe0b6ed32e461e1ab10a75b2df3885

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Atmospheric Science

In Environmental Engineering

In Water Science and Technology

Citations in the corpus

Comparing expert judgement and numerical criteria for hydrograph evaluation (2015, background): “…did not include some of the criteria recently proposed in the literature (Wang and Melesse 2005, Mobley et al. 2012, Pushpalatha et al. 2012, Willmott et al. 2012, Bardsley 2013, Jagupilla et al. 2014), nor criteria based on information theory (see e.g.Weijs et al. 2010, Pechlivanidis et al. 2012).”
Effectiveness of Entropy Weight Method in Decision-Making (2020, unknown): “Subsequently, the decision-making result is distorted [14, 15].”
Estimating information entropy for hydrological data: One‐dimensional case (2014, background): “…2009b], input variable selection for seasonal and interannual rainfall probabilistic forecasts [Sharma et al., 2000; Sharma, 2000a, 2000b; Fernando et al., 2009; May et al., 2008], and quantification of the skill of ensemble forecasts [Weijs et al., 2010a, 2010b; Weijs and van de Giesen, 2011].”
Use of an entropy‐based metric in multiobjective calibration to improve model performance (2014, background): “Recent work has proposed the concept of a diagnostic evaluation approach rooted in information theory [Gupta et al., 2008; Weijs et al., 2010].”
Characterising performance of environmental models (2013, background): “Weijs et al. (2010) endorse the use of information theory and corresponding scores for evaluating models, arguing that such approaches maximise the information extracted from observations.”
Generalization of the Ignorance Score: Continuous Ranked Version and Its Decomposition (2012, methodology): “Consequently, it can also be decomposed into ranked versions of the binary components like in (5), as mentioned by Weijs et al. (2010a): RIGN 5 k IGNk 5 k RELk 2 k RESk 1 k UNCk.”