Calibrated asymmetric surrogate losses

By Clayton D. Scott. 2012

99 citations recorded by the source corpus.

Surrogate losses underlie numerous state-of-the-art binary classification algorithms, such as support vector machines and boosting. The impact of a surrogate loss on the statistical performance of an algorithm is well-understood in symmetric classification settings, where the misclassification costs are equal and the loss is a margin loss. In particular, classification-calibrated losses are known to imply desirable properties such as consistency. While numerous efforts have been made to extend surrogate loss-based algorithms to asymmetric settings, to deal with unequal misclassification costs or training data imbalance, considerably less attention has been paid to whether the modified loss is still calibrated in some sense. This article extends the theory of classification-calibrated losses to asymmetric problems. As in the symmetric case, it is shown that calibrated asymmetric surrogate losses give rise to excess risk bounds, which control the expected misclassification cost in terms of the excess surrogate risk. This theory is illustrated on the class of uneven margin losses, and the uneven hinge, squared error, exponential, and sigmoid losses are treated in detail.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Fairness-aware Classification: Criterion, Convexity, and Bounds (2018, neutral): “[Scott 2012] Scott, C. 2012.”
General Bayesian updating and the loss-likelihood bootstrap (2017, methodology): “Both of our methods can easily be adapted to cost-sensitive classification problems, by use of an asymmetric hinge loss (Scott, 2012).”
Convex Calibration Dimension for Multiclass Loss Matrices (2014, methodology): “Variants of standard 0-1 binary classiﬁcation have also been studied; for example, Yuan and Wegkamp [32] studied consistency for the problem of binary classiﬁcation with a reject option, and Scott [25] studied calibrated surrogates for cost-sensitive binary classiﬁcation.”
Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings (2015, background): “…Bartlett et al. (2006), Tewari and Bartlett (2005), Zhang (2004), Steinwart (2007), Reid and Williamson (2011), Calauzènes et al. (2012, 2013), Scott (2012), Duchi et al. (2013), Menon and Williamson (2014), Agarwal and Agarwal (2015), Frongillo and Kash (2015) and Ramaswamy and Agarwal…”
Learning with Noisy Labels (2013, background): “The following lemma [Scott, 2012] shows that in fact for any α-weighted 0-1 loss, the minimizer thresholds η(x) at α. Lemma 8 (α-weighted Bayes optimal [Scott, 2012]).”
Class-Weighted Classification: Trade-offs and Robust Approaches (2020, methodology): “While previous cost weighting setups have normalized the sum of weights Scott (2012), our normalization scheme is computed with respect to prior probabilities on each class as well, and consequently we explicitly include 𝑞 0 , 𝑞 1 in our bound.”
Ieee Transactions on Pattern Analysis and Machine Intelligence Classification with Noisy Labels by Importance Reweighting (unknown): “This modification is based on the asymmetric classification-calibrated results [12] and cannot be used arXiv:1411.7718v2 [stat.”