Increasing the Reliability of Reliability Diagrams

By Jochen Bröcker, Leonard A. Smith. 2007

309 citations recorded by the source corpus.

Subfields: Atmospheric Science, Global and Planetary Change, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Atmospheric Science

In Global and Planetary Change

In Management Science and Operations Research

Citations in the corpus

Verified Uncertainty Calibration (2019, methodology): “Besides the calibration error, prior work also uses the Hosmer-Lemeshov test [37] and reliability diagrams [4, 38] to evaluate calibration.”
Impact of Stochastic Physics in a Convection-Permitting Ensemble (2012, methodology): “The diagram was constructed following recommendations of Bröcker and Smith (2007); observation errors are neglected.”
A bias‐corrected decomposition of the Brier score (2012, background): “Usually all nk exceed 1 because small nk are often eradicated by relabelling distinct forecasts with a common forecast value (e.g. Bröcker and Smith, 2007), although this will typically change the limiting values REL∞ and RES∞ being estimated.”
Combining probability forecasts (2010, background): “Genest and Zidek (1986), Wallsten et al. (1997), Clemen and Winkler (2007) and Primo et al. (2009) provide excellent reviews.”
Estimation Methods for Nonhomogeneous Regression Models: Minimum Continuous Ranked Probability Score versus Maximum Likelihood (2018, background): “The 95% consistency interval shown as gray area, are derived similar to Bröcker and Smith (2007) and show the expected binwise sampling variations.”
Evaluating model calibration in classification (2019, methodology): “In Figure 4, we present our variant of the classical reliability diagram inspired by Bröcker and Smith (2007).”
Calibration tests in multi-class classification: A unifying framework (2019, methodology): “data sets are resampled under the assumption that the model is calibrated by sampling labels from resampled predictions (Bröcker and Smith, 2007; Vaicenavicius et al., 2019).”
Probability Aggregation Methods in Geoscience (2012, background): “Reliability diagrams plot the empirical event frequency against the aggregated probabilities (Bröcker and Smith 2007).”
Probabilistic quantitative precipitation forecasting using Ensemble Model Output Statistics (2013, methodology): “We adopt the paradigm formulated by Bröcker and Smith (2008), that post-processing should make optimal use of the information contained in the ensemble without relying on any assumption about ensemble forecasts being draws from some unknown distributions. Our aim is to develop a model that is of com”
An analog ensemble for short-term probabilistic solar power forecast (2015, methodology): “The vertical bars represent the confidence intervals [42,43] and are calculated with a quantile function for a binomial distribution.”
A tutorial on calibration measurements and calibration models for clinical prediction models (2020, background): “Reliability diagram 25,26,50,51 Allows for visualization of regions of miscalibration and the “direction” of miscalibration (ie, underestimation, overestimation) Not a continuous graph.”
Probabilistic Forecast Calibration Using ECMWF and GFS Ensemble Reforecasts. Part II: Precipitation (2008, methodology): “Also, 5% and 95% confidence intervals were placed on the reliability curves, with the confidence intervals estimated from a 1000-member block bootstrap sample [following Efron and Tibshirani (1993) and Hamill (1999), and similar to the method in Bröcker and Smith (2007)].”