Probabilistic forecasts, calibration and sharpness

By Tilmann Gneiting, Fadoua Balabdaoui, Adrian E. Raftery. 2007

1,916 citations recorded by the source corpus.

Subfields: Atmospheric Science, Environmental Engineering, Global and Planetary Change, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Atmospheric Science

In Environmental Engineering

In Global and Planetary Change

In Probabilistic & Bayesian Methods

Citations in the corpus

Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, background): “Specifically, there can exist a trivial predictor that ignores the input example and achieves perfect calibration by predicting randomly according to the marginal distribution of the labels (Gneiting et al., 2007).”
Dark Experience for General Continual Learning: a Strong, Simple Baseline (2020, background): “weather forecasting [5] or econometric analysis [13]); therefore, calibration represents an appealing property for any CL system aiming for employment outside of a laboratory environment.”
Verified Uncertainty Calibration (2019, background): “We often want to minimize the MSE subject to a calibration budget [5, 25].”
Measuring Calibration in Deep Learning (2019, background): “For example, Brier score measures the mean squared difference between the predicted probability and the actual outcome (Gneiting and Raftery, 2007).”
Accurate Uncertainties for Deep Learning Using Calibrated Regression (2018, background): “Although most work on calibration focuses on classiﬁcation, Gneiting et al. (2007) proposed several deﬁnitions of calibration for continuous variables.”
The Problem of Infra-marginality in Outcome Tests for Discrimination (2016, positive): “absence of discrimination, all drivers who have positive utility of carrying contraband are equally likely to do so. 5 meaning that the predictions are statistically consistent with the observations [Gneiting et al., 2007]: when ocers think a driver has probability pof having contraband, p-percent o”
Forecast combinations: An over 50-year review (2022, background): “Gneiting and Ranjan (2013) highlighted that the generalized pooling strategy may fail to be sufficiently flexibly dispersive for calibration.”
Probabilistic quantitative precipitation forecasting using Ensemble Model Output Statistics (2013, background): “These two properties should be the goal of probabilistic forecasting (Gneiting et al., 2007), and so it is natural to use strictly proper scoring rules as loss functions for training algorithms.”
Electricity price forecasting: A review of the state-of-the-art with a look into the future (2014, background): “do not perform formal statistical tests, however, Gneiting et al. (2007) argue that the null hypothesis of no difference in predictive performance can be easily tested given the CRPS values.”
Forecasting with DSGE Models (2010, neutral): “Moreover, as Geweke and Amisano (2009) and Gneiting, Balabdaoui, and Raftery (2007) note, the assessment of a predictive distribution on the basis of its density and the observed data only is consistent with the prequential approach.”
Probabilistic Forecast Calibration Using ECMWF and GFS Ensemble Reforecasts. Part II: Precipitation (2008, background): “Calibration here refers to the statistical adjustment of numerical forecasts to produce probabilistic forecasts that are as sharp as possible while remaining reliable (Wilks 2006, 258–259; Gneiting et al. 2007).”
Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules (2011, methodology): “The probability integral transform (PIT) histogram is the primary diagnostic tool for calibration checks (Diebold, Gunther, and Tay 1998; Corradi and Swanson 2006b; Gneiting, Balabdaoui, and Raftery 2007; Laio and Tamea 2007).”
Bias Correction and Bayesian Model Averaging for Ensemble Forecasts of Surface Wind Direction (2010, methodology): “Instead, we use circular analogs of the absolute error and the continuous ranked probability score, as introduced by Grimit et al. (2006).”
Bayesian Nonparametric Calibration and Combination of Predictive Distributions (2015, methodology): “A prominent, critically important issue is that predictive distributions ought to be calibrated (Dawid, 1984; Kling and Bessler, 1989; Diebold et al., 1998; Gneiting et al., 2007; Mitchell and Wallis, 2011).”
Spatial dynamic factor analysis (2008, methodology): “Further evaluation of the predictive performance can be made using criteria proposed by Gneiting, Balabdaoui, and Raftery (2007).”
Short‐Term Wind Speed Forecasting for Power System Operations (2012, background): “To evaluate the quality of probabilistic forecasts, Gneiting et al. (2007) pointed out that the goal of probabilistic forecasting is to maximize the sharpness of the predictive distribution subject to calibration.”
Assessing the Calibration of High-Dimensional Ensemble Forecasts Using Rank Histograms (2013, methodology): “This is a compromise between flexible adaptation to seasonal changes on the one hand and gathering sufficient data to permit stable model fitting on the other hand, see, for example, Gneiting et al. (2005) and Raftery et al.”