The Comparison and Evaluation of Forecasters.

By Morris H. Degroot, Stephen E. Fienberg. 1983

Subfields: Evaluation & Benchmarks, Time-Series & Forecasting

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

In Time-Series & Forecasting

Citations in the corpus

The Right Tool for the Job: Matching Model and Instance Complexities (2020, background): “One way to mitigate this concern is to use calibration, which encourages the conﬁdence level to correspond to the probability that the model is correct (DeGroot and Fienberg, 1983).”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, methodology): “Firstly, we shall examine calibration [12, 13], a frequentist notion of uncertainty which measures the discrepancy between subjective forecasts and (empirical) long-run frequencies.”
Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, background): “Brier score is also related to the ECE in that it is an upper bound of the calibration error by the classic calibration-refinement decomposition (DeGroot and Fienberg, 1983).”
Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation (2017, background): “ch we exploit in Section 2.3.3 to detect differences between the mimic and outcome models. 2.3.2 Calibrating model inputs. Calibration is the process of matching predicted and empirical probabilities [14, 28]. If a risk score is well-calibrated, the relationship between the risk score and empirical ”
Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data (2019, methodology): “Here, the predictive uncertainty is calibrated using the reliability diagram [59].”
BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning (2020, background): “Several measures have been proposed to assess the quality of uncertainty estimates, such as calibration (Dawid, 1982; Degroot & Fienberg, 1983).”
Verified Uncertainty Calibration (2019, methodology): “The importance of this notion of calibration has been emphasized in personalized medicine [1], meteorological forecasting [2, 3, 4, 5, 6] and natural language processing applications [7, 8].”
Measuring Calibration in Deep Learning (2019, background): “Expected Calibration Error was crafted to mirror reliability diagrams, which are structured around binary classi-ﬁcation such as rain vs not rain (DeGroot and Fienberg, 1983).”
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019, methodology): “We refer to DeGroot & Fienberg (1983) for the decomposition of Brier score into calibration and reﬁnement for classiﬁcation and to (Br¨ocker, 2009) for the general decomposition for any proper scoring rule.”
Posterior calibration and exploratory analysis for natural language processing models (2015, methodology): “5 Applying this factorization to the Brier score leads to the calibrationrefinement decomposition (DeGroot and Fienberg, 1983), in terms of expectations with respect to the prediction strength distribution P(q):”
Obtaining Well Calibrated Probabilities Using Bayesian Binning (2015, background): “Figure 1 shows a hypothetical example of a reliability curve (DeGroot and Fienberg 1983; Niculescu-Mizil and Caruana 2005), which displays the calibration performance of a prediction method.”
Practical Deep Learning with Bayesian Principles (2019, neutral): “We also show calibration curves [7] in Figures 1 and 14.”
PAC Confidence Predictions for Deep Neural Network Classifiers (2020, methodology): “We ﬁrst introduce the deﬁnition of calibration (DeGroot & Fienberg, 1983; Zadrozny & Elkan, 2002; Park et al., 2020b)— i.e., what we mean for a predicted conﬁdence to be “correct”.”
Comparison Methods for Stochastic Models and Risks (2005, background): “Also, the work on calibrating probability forecasters (cf. DeGroot and Fienberg 1983) is relevant, as is more detail on elicitation (cf. Kadane and Winkler 1988).”
An experimental comparison of performance measures for classification (2009, background): “It is usually a measure of the reliability of the prediction (DeGroot and Fienberg, 1982).”
Local Temperature Scaling for Probability Calibration (2020, methodology): “As it is difficult to work directly with this definition to assess miscalibration, we extend several visual and quantitative metrics [11, 53, 54, 56, 57], which have previously been proposed in the context of classification.”
Auditing Black-Box Models Using Transparent Model Distillation With Side Information (2017, background): “Consider the risk score reliability diagrams [15, 37] in Figure 1.”
Non-Parametric Calibration for Classification (2019, background): “Originally developed in the context of forecasting (Murphy, 1973, DeGroot and Fienberg, 1983), uncertainty calibration has seen an increased interest in recent years (Naeini et al., 2015, Guo et al., 2017, Vaicenavicius et al., 2019), partly because of the popularity of CNNs which generally lack an…”