Predicting accurate probabilities with a ranking loss

By Aditya Krishna Menon, Xiaoqian Jiang, Shankar Vembu, Charles Peter Elkan, Lucila Ohno-Machado. 2012

79 citations recorded by the source corpus.

In many real-world applications of machine learning classifiers, it is essential to predict the probability of an example belonging to a particular class. This paper proposes a simple technique for predicting probabilities based on optimizing a ranking loss, followed by isotonic regression. This semi-parametric technique offers both good ranking and regression performance, and models a richer set of probability distributions than statistical workhorses such as logistic regression. We provide experimental results that show the effectiveness of this technique on real-world applications of probability prediction.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Learning Optimized Risk Scores (2016, background): “In particular, the logistic loss is a strictly proper loss (Reid and Williamson, 2010; Ertekin and Rudin, 2011) which yields calibrated risk estimates under the parametric assumption that the true risk can be modeled using a logistic link function (see Menon et al., 2012).”
Estimating treatment effect heterogeneity in randomized program evaluation (2013, methodology): “While it is known that the SVM does not return explicit probability estimates [Lin (2002), Lee, Lin and Wahba (2004)], we follow work that transforms the values Ŵi(t) to approximate the underlying probability [Franc, Zien and Schölkopf (2011), Sollich (2002), Platt (1999), Menon et al. (2012)].”
Obtaining Well Calibrated Probabilities Using Bayesian Binning (2015, methodology): “In order to mitigate this pitfall, Menon et. al (Menon et al. 2012) proposed a new isotonic based calibration method using a combination of optimizing AU C as a ranking loss measure, plus isotonic regression for building an accurate ranking model.”
Differential privacy based on importance weighting (2013, methodology): “Other methods of estimating well-calibrated conditional probabilities (Zadrozny and Elkan 2001; Kanamori et al. 2009; Menon et al. 2012) can be used also, if modified to guarantee differential privacy.”
Binary Classifier Calibration: Non-parametric approach (2014, methodology): “In order to mitigate this pitfall, Menon et. all (13) proposed using a combination of optimizing AUC as a ranking loss measure, plus isotonic regression for building a ranking model.”
Optimal Thresholding of Classifiers to Maximize F1 Measure (2014, background): “Note that square loss is a proper loss function and yields calibrated probabilistic predictions [12].”
Calibrated Structured Prediction (2015, methodology): “Non-binary methods typically involve training one-vs-all predictors [12] and include extensions to ranking losses [19] and combinations of estimators [20].”
Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labels (2017, background): “Removal by rank mitigates sensitivity to probability estimation and exploits the reduced complexity of learning to rank over probability estimation (Menon et al., 2012).”