Kullback–Leibler Divergence as a Forecast Skill Score with Classic Reliability–Resolution–Uncertainty Decomposition

By Steven V. Weijs, Ronald R. P. van Nooijen, N. V. D. Giesen. 2010

97 citations recorded by the source corpus.

Abstract This paper presents a score that can be used for evaluating probabilistic forecasts of multicategory events. The score is a reinterpretation of the logarithmic score or ignorance score, now formulated as the relative entropy or Kullback–Leibler divergence of the forecast distribution from the observation distribution. Using the information–theoretical concepts of entropy and relative entropy, a decomposition into three components is presented, analogous to the classic decomposition of the Brier score. The information–theoretical twins of the components uncertainty, resolution, and reliability provide diagnostic information about the quality of forecasts. The overall score measures the information conveyed by the forecast. As was shown recently, information theory provides a sound framework for forecast verification. The new decomposition, which has proven to be very useful for the Brier score and is widely used, can help acceptance of the logarithmic score in meteorology.

Subfields: Atmospheric Science, Global and Planetary Change

View Paper: https://www.semanticscholar.org/paper/206133793c44d1a86d5aa7dc7b1047b815c706a6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Atmospheric Science

In Global and Planetary Change

Citations in the corpus

Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings (2015, background): “In DeGroot and Fienberg (1983) or Weijs et al. (2010), a decomposition of the divergence score was presented.”
Why hydrological predictions should be evaluated using information theory (2010, methodology): “Section 2 introduces the divergence score or logarithmic score as a measure of forecast quality (Weijs et al., 2010), which will be used as a starting point for the interpretations in Sects.”
Estimating epistemic and aleatory uncertainties during hydrologic modeling: An information theoretic approach (2013, methodology): “Close to the focus of this paper, information theory has been applied for model evaluation and uncertainty analysis as far back as the 1970s [e.g., Amorocho and Espildor, 1973; Chapman, 1986; Weijs et al., 2010a, 2010b; Weijs and van de Giesen, 2011; Abebe and Price, 2003; Pokhrel and Gupta, 2010].”
Generalization of the Ignorance Score: Continuous Ranked Version and Its Decomposition (2012, methodology): “Consequently, it can also be decomposed into ranked versions of the binary components like in (5), as mentioned by Weijs et al. (2010a): RIGN 5 k IGNk 5 k RELk 2 k RESk 1 k UNCk.”