Binary Classifier Calibration: Non-parametric approach

By Mahdi Pakdaman Naeini, Gregory F. Cooper, Milos Hauskrecht. 2014

18 citations recorded by the source corpus.

Accurate calibration of probabilistic predictive models learned is critical for many practical prediction and decision-making tasks. There are two main categories of methods for building calibrated classifiers. One approach is to develop methods for learning probabilistic models that are well-calibrated, ab initio. The other approach is to use some post-processing methods for transforming the output of a classifier to be well calibrated, as for example histogram binning, Platt scaling, and isotonic regression. One advantage of the post-processing approach is that it can be applied to any existing probabilistic classification model that was constructed using any machine-learning method. In this paper, we first introduce two measures for evaluating how well a classifier is calibrated. We prove three theorems showing that using a simple histogram binning post-processing method, it is possible to make a classifier be well calibrated while retaining its discrimination capability. Also, by casting the histogram binning method as a density-based non-parametric binary classifier, we can extend it using two simple non-parametric density estimation methods. We demonstrate the performance of the proposed calibration methods on synthetic and real datasets. Experimental results show that the proposed methods either outperform or are comparable to existing calibration methods.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Learning Optimized Risk Scores (2016, methodology): “CAL is the expected calibration error over the reliability diagram (Naeini et al., 2014).”
Verified Uncertainty Calibration (2019, background): “In particular, the number of samples required to calibrate scales linearly with the number of distinct probabilities the model can output, B [14], which can 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.”
Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift (2022, background): “Calibration has been widely studied in machine learning [Naeini et al., 2014, Guo et al., 2017, Kumar et al., 2019], and applications such as meteorology [Murphy, 1973, DeGroot and Fienberg, 1983, Gneiting and Raftery, 2005], fairness [Hebert-Johnson et al., 2018], and health-care [Jiang et al.,…”