Transforming classifier scores into accurate multiclass probability estimates

By Bianca Zadrozny, Charles Peter Elkan. 2002

1,705 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

Citations in the corpus

RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, methodology): “Since the model is calibrated using isotonic regression (Zadrozny and Elkan, 2002),5 we can meaningfully interpret the score as a probability of toxicity.”
Association of an Electroencephalography-Based Risk Score With Seizure Probability in Hospitalized Patients (2017, neutral): “0 1 2 3 4 5 >6 Probable risk of Sz, %b 5 12 27 50 73 88 >95 Actual prevalence of Sz, % (95% CI)c 3 (2-3) 12 (10-13) 34 (31-37) 52 (46-57) 71 (63-78) 84 (71-99) 92 (77-100)”
Verified Uncertainty Calibration (2019, methodology): “We recalibrated each class separately as in [13], using B bins per class, and evaluated calibration using the marginal calibration error (Definition 2.”
Measuring Calibration in Deep Learning (2019, methodology): “, 1999) and the non-parametric approach of isotonic regression (Zadrozny and Elkan, 2002).”
Calibrating Deep Convolutional Gaussian Processes (2018, methodology): “Popular calibration techniques include Platt scaling [24] and isotonic regression [30].”
Posterior calibration and exploratory analysis for natural language processing models (2015, methodology): “Finally, it may be interesting to pursue recalibration methods, which readjust a non-calibrated model’s predictions to be calibrated; recalibration methods have been developed for binary (Platt, 1999; Niculescu-Mizil and Caruana, 2005) and multiclass (Zadrozny and Elkan, 2002) classification setting”
Obtaining Well Calibrated Probabilities Using Bayesian Binning (2015, methodology): “The existing approaches developed for this purpose include histogram binning, Platt scaling, or isotonic regression (Platt 1999; Zadrozny and Elkan 2001; 2002).”
Accurate Uncertainties for Deep Learning Using Calibrated Regression (2018, background): “They admit extensions to the multi-class setting (Zadrozny and Elkan, 2002) and to structured prediction (Kuleshov and Liang, 2015).”
Learning Cumulatively to Become More Knowledgeable (2016, methodology): “As the decision score produced by SVM is not a probability distribution, several techniques have been proposed to convert a raw decision score to calibrated probabilistic outputs [2, 12, 17, 27, 41].”
To Trust Or Not To Trust A Classifier (2018, background): “One related line of work is that of conﬁdence calibration, which transforms classiﬁer outputs into values that can be interpreted as probabilities, e.g. [44, 58, 40, 24].”
Predicting good probabilities with supervised learning (2005, methodology): “Platt Scaling: a method for transforming SVM outputs from [−∞,+∞] to posterior probabilities (Platt, 1999) Isotonic Regression: the method used by Zadrozny and Elkan (2002; 2001) to calibrate predictions from boosted naive bayes, SVM, and decision tree models Platt Scaling is most effective when…”
Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings (2018, methodology): “In machine learning, many different methods have been proposed for enhancing calibration of classifiers including Platt’s scaling (Platt, 1999), Isotonic regression (Zadrozny & Elkan, 2002), and Bayesian binning (Naeini et al., 2015), to name a few.”
Binary Classifier Calibration: Bayesian Non-Parametric Approach (2014, methodology): “This paper presents two new non-parametric methods for calibrating outputs of binary classification models: a method based on the Bayes optimal selection and a method based on the Bayesian model averaging.”
Multicategory Support Vector Machines, Theory, and Application to the Classiﬁcation of Microarray Data and Satellite Radiance Data (2002, methodology): “…have been attempts to map outputs of classi ers to conditional probabilities for various classi - cation methods, including the SVM, in multiclass problems (see Zadrozny and Elkan 2002; Passerini, Pontil, and Frasconi 2002; Price, Knerr, Personnaz, and Dreyfus 1995; Hastie and Tibshirani 1998).”
Outlier Analysis (2013, background): “in the supervised scenario have been discussed in [495].”
PAC Confidence Predictions for Deep Neural Network Classifiers (2020, methodology): “Histogram binning is a non-parametric approach that sorts the data into ﬁnitely many bins and rescales the conﬁdences per bin (Zadrozny & Elkan, 2001; 2002; Naeini et al., 2015).”