Obtaining Well Calibrated Probabilities Using Bayesian Binning

By Mahdi Pakdaman Naeini, Gregory F. Cooper, Milos Hauskrecht. 2015

1,951 citations recorded by the source corpus.

Learning probabilistic predictive models that are well calibrated is critical for many prediction and decision-making tasks in artificial intelligence. In this paper we present a new non-parametric calibration method called Bayesian Binning into Quantiles (BBQ) which addresses key limitations of existing calibration methods. The method post processes the output of a binary classification algorithm; thus, it can be readily combined with many existing classification algorithms. The method is computationally tractable, and empirically accurate, as evidenced by the set of experiments reported here on both real and simulated datasets.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

Citations in the corpus

A Review of Sparse Expert Models in Deep Learning (2022, methodology): “This work measured calibration by the Expected Calibration Error (Naeini et al., 2015) which is the absolute deviation between the predicted probability and average accuracy, after binning examples by their predicted probability.”
On Calibration of Modern Neural Networks (2017, methodology): “Bayesian Binning into Quantiles (BBQ) (Naeini et al., 2015) is a extension of histogram binning using Bayesian 1 This is in contrast with the setting in Section 2, in which the model produces both a class prediction and confidence.”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, neutral): “Please refer to [48, 19] for more details.”
Dark Experience for General Continual Learning: a Strong, Simple Baseline (2020, background): “[31] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht.”
Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks (2020, methodology): “To answer this, we use a standard metric (Naeini et al., 2015; Guo et al., 2017): the expected calibration error (ECE).”
BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning (2020, methodology): “Appendix D also uses Expected Calibrated Error (ECE) (Guo et al., 2017; Naeini et al., 2015) as an uncertainty metric.”
Verified Uncertainty Calibration (2019, methodology): “1 and `1 calibration errors are also used in the literature [9, 23, 24].”
Measuring Calibration in Deep Learning (2019, background): “ECE as framed in Naeini et al. (2015) leaves ambiguity in both its binning implementation and how to compute calibration for multiple classes.”
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019, background): “Expected Calibration Error (ECE) ↓ Measures predicted probability accuracy (Naeini et al., 2015).”
Practical Deep Learning with Bayesian Principles (2019, methodology): “We compare the probabilities using the following metrics: validation negative log-likelihood (NLL), area under ROC (AUROC) and expected calibration curves (ECE) [40, 15].”
Analyzing the role of model uncertainty for electronic health records (2019, background): “• held-out negative log-likelihood (all tasks), • ECE (all tasks) [23], and • adaptive calibration error (ACE) (all tasks) [24].”
Mitigating bias in calibration error estimation (2020, methodology): “Given that the selection of the training objectives and the justification of a recalibration technique is predicated on the measurement of the calibration 1 Naeini et al. (2015) introduce ECE as an acronym for Expected Calibration Error.”
Mitigating Neural Network Overconfidence with Logit Normalization (2022, methodology): “In practice, Expected Calibration Error (ECE) (Naeini et al., 2015) is commonly used to measure the calibration performance from ﬁnite samples.”
L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models (2023, methodology): “Following recent work (Liang et al., 2022), we evaluate model calibration using expected calibration error (ECE) (Naeini et al., 2015; Guo et al., 2017).”
Dynamically Weighted Balanced Loss: Class Imbalanced Learning and Confidence Calibration of Deep Neural Networks (2021, methodology): “Calibration Metrics: We evaluate the calibration performance based on Expected Calibration Error (ECE) [50], Maximum Calibration Error (MCE) and Brier Score (BS) [51].”
PAC Confidence Predictions for Deep Neural Network Classifiers (2020, background): “ove DNN conﬁdences (Guo et al., 2017). Histogram binning is a non-parametric approach that sorts the data into ﬁnitely many bins and rescales the conﬁdences per bin (Zadrozny & Elkan, 2001; 2002; Naeini et al., 2015). However, traditional approaches do not provide theoretical guarantees on the p”
Non-Parametric Calibration for Classification (2019, methodology): “More recently, Bayesian binning into quantiles (Naeini et al., 2015) and beta calibration (Kull et al., 2017a) for binary classiﬁcation and temperature scaling (Guo et al., 2017) for multi-class problems were proposed.”
Calibration tests in multi-class classification: A unifying framework (2019, background): “, 2015), the maximum calibration error (MCE) (Naeini et al., 2015), and the kernel-based maximum mean calibration error (MMCE) (Kumar et al.”
Transformers Can Do Bayesian Inference (2021, methodology): “We ﬁnd that on our data they have lower Expected Calibration Error (ECE, using 100 bins (Naeini et al., 2015)) even than standard BNNs and ensemble methods such as XGBoost, the two baseline methods that best capture uncertainty.”