Reliability, Sufficiency, and the Decomposition of Proper Scores

By J. Brocker. 2008

205 citations recorded by the source corpus.

Scoring rules are an important tool for evaluating the performance of probabilistic forecasting schemes. In the binary case, scoring rules (which are strictly proper) allow for a decomposition into terms related to the resolution and to the reliability of the forecast. This fact is particularly well known for the Brier Score. In this paper, this result is extended to forecasts for finite–valued targets. Both resolution and reliability are shown to have a positive effect on the score. It is demonstrated that resolution and reliability are directly related to forecast attributes which are desirable on grounds independent of the notion of scores. This finding can be considered an epistemological justification of measuring forecast quality by proper scores. A link is provided to the original work of DeGroot and Fienberg (1982), extending their concepts of sufficiency and refinement. The relation to the conjectured sharpness principle of Gneiting et al. (2005a) is elucidated.

Subfields: Atmospheric Science, Global and Planetary Change, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Atmospheric Science

In Global and Planetary Change

In Management Science and Operations Research

Citations in the corpus

Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, background): “Proper scoring rules are related to ECE in that it is an upper bound of the calibration error by the classic calibration-refinement decomposition [10].”
Verified Uncertainty Calibration (2019, background): “Other notions of multiclass calibration include joint calibration (which requires the entire probability vector to be calibrated) [2, 6] and event-pooled calibration [16].”
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019, methodology): “We refer to DeGroot & Fienberg (1983) for the decomposition of Brier score into calibration and reﬁnement for classiﬁcation and to (Br¨ocker, 2009) for the general decomposition for any proper scoring rule.”
Posterior calibration and exploratory analysis for natural language processing models (2015, background): “These two loss functions are instances of proper scoring rules (Gneiting and Raftery, 2007; Bröcker, 2009).”
Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision (2019, methodology): “As an absolute measure of uncertainty estimation quality, we therefore also evaluate the methods in terms of calibration [5, 46].”
Dynamically Weighted Balanced Loss: Class Imbalanced Learning and Confidence Calibration of Deep Neural Networks (2021, background): “Recall that entropy of a probability assignment is a measure of inherent uncertainty [47].”
Forecast verification for extreme value distributions with an application to probabilistic peak wind prediction (2012, background): “An important characteristic of a qualified scoring rule is its propriety (Murphy, 1973; Gneiting and Raftery, 2007; Bröcker and Smith, 2007). A scoring rule is (strictly) proper if the expected score for an observation Y is optimized if (and only if) the true distribution of Y is issued as the forec”
A bias‐corrected decomposition of the Brier score (2012, background): “In fact, all proper scores can be decomposed into reliability, resolution and uncertainty terms (Bröcker, 2009).”
The role of the information set for forecasting—with applications to risk management (2014, background): “The second part of the corollary extends results by Bröcker (2009) and DeGroot and Fienberg (1983) from finite to general real state space.”
Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift (2022, methodology): “Formally, we look at joint calibration [Murphy, 1973, Brocker, 2009] where a model f is calibrated with respect to a distribution P if for all x ∈ X , y ∈ [ K ] : The following proposition says that if f std and f rob are calibrated on P id , then f ens has lower error on P id than any other way of…”
Introducing independent patterns into the Stochastically Perturbed Parametrization Tendencies (SPPT) scheme (2017, background): “It is a proper score and is therefore sensitive to both reliability and resolution (Bröcker, 2009):”
Revisiting the Calibration of Modern Neural Networks (2021, methodology): “For example, if we take all data points x for which the model predicts [f(x)]y = 0.3, we expect 30% of them to indeed take on the label y. Formally, the model f is said to be calibrated if (Bröcker, 2009) ∀p ∈ ∆: P (Y = y | f(X) = p) = py.”
Calibration tests in multi-class classification: A unifying framework (2019, methodology): “2 Strong notion According to the more general calibration definition of Bröcker (2009); Vaicenavicius et al. (2019), a probabilistic model g is calibrated if for almost all inputs x the prediction g(x) is equal to the distribution of class Y given prediction g(X) = g(x).”