Predicting good probabilities with supervised learning

By Alexandru Niculescu-Mizil, R. Caruana. 2005

2,099 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Statistics and Probability, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Evaluation & Benchmarks

Citations in the corpus

Consistent Accelerated Inference via Confident Adaptive Transformers (2021, background): “A large amount of research has been dedicated towards calibrating the model posterior, pθ(ŷn+1|xn+1), such that the accuracy, yn+1 = ŷn+1, is indeed equal to the estimated probability (Niculescu-Mizil and Caruana, 2005; Gal and Ghahramani, 2016; Guo et al., 2017).”
Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions (2018, background): “(3)We assume scores have been calibrated to be interpreted as probabilities, though this is non-trivial in practice [129].”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, positive): “We found that this procedure gives a more effective visualization of the actual conﬁdence distribution of DNN predictions than the standard reliability diagrams used in Guo et al. [19] and Niculescu-Mizil and Caruana [50].”
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning (2018, methodology): “Hence, we plot reliability diagrams to visualize the calibration of our credibility [88].”
Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation (2017, methodology): “Specifically, we learn a nonlinear transformation of the risk score (the blue line in Figure 6 in the Appendix), similar to isotonic regression [28], to make the risk scores and outcomes linearly related on a scale of choice.”
On Fairness and Calibration (2017, methodology): “See [30] and [18] for empirical comparisons of these methods.”
Interpretable classification models for recidivism prediction (2015, methodology): “ Zeng, Ustun, and Rudin In addition to predictive accuracy, we also examine the risk calibration of the models. Figure 2 show the risk calibration for arrest, constructed from the method presented in Niculescu-Mizil and Caruana (2005). The calibration plots for all other problems are included in App”
Measuring Calibration in Deep Learning (2019, methodology): “The original formulation of Platt scaling for neural networks (Niculescu-Mizil and Caruana, 2005) involves learning scalar parameters a, b ∈ R on a held-out validation set and then computing calibrated probabilities p̂i given the uncalibrated logits vector zi on the test set as p̂i = σ(azi + b).”
Calibrating Deep Convolutional Gaussian Processes (2018, methodology): “Classiﬁers based on Deep Neural Networks ( DNN s) have been shown to be well-calibrated [23].”
Posterior calibration and exploratory analysis for natural language processing models (2015, methodology): “Previous studies that assess calibration in supervised machine learning models (Niculescu-Mizil and Caruana, 2005; Bennett, 2000) calculate label frequencies by dividing the prediction space into deciles or other evenly spaced bins—e.g. q ∈ [0 , 0 .”
Obtaining Well Calibrated Probabilities Using Bayesian Binning (2015, methodology): “This method only requires that the mapping function be isotonic (monotonically increasing) (Niculescu-Mizil and Caruana 2005).”
Accurate Uncertainties for Deep Learning Using Calibrated Regression (2018, methodology): “In the binary classification setting, Platt scaling (Platt, 1999) and isotonic regression (Niculescu-Mizil and Caruana, 2005) are effective and widely used to perform recalibration.”
Enhancing Evolutionary Couplings with Deep Convolutional Neural Networks (2017, background): “The key insights of modern machine learning include both the power of automatic feature selection and the ability to integrate data sources, as well as the ability to leverage and encode contextual data and to convert inputs of arbitrary units into well-calibrated probabilities (Cho et al., 2016; Jo”
Ad click prediction: a view from the trenches (2013, methodology): “We can ﬁnd such a mapping using isotonic regression, which computes a weighted least-squares ﬁt to the input data subject to that constraint (see, e.g., [27, 23]).”
To Trust Or Not To Trust A Classifier (2018, background): “One related line of work is that of conﬁdence calibration, which transforms classiﬁer outputs into values that can be interpreted as probabilities, e.g. [44, 58, 40, 24].”
Design Challenges and Misconceptions in Named Entity Recognition (2009, methodology): “Furthermore, both beamsearch and Viterbi require transforming the predictions of the classi-149 ﬁers to probabilities as discussed in (Niculescu-Mizil and Caruana, 2005), incurring additional time overhead.”
Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting (2019, background): “As shown in [61], shallow networks are well-calibrated and [62] further argue that a major reason for miscalibration is the use of a softmax loss typical for multi-class problems.”
Simple Rules for Complex Decisions (2017, background): “We opt for lasso instead of random forest for this prediction task because the laer, while very good for classication, is known to suer from poor calibration [26], which can in turn yield biased estimates of a policy’s eects.”