Composite Binary Losses

By Mark D. Reid, Robert C. Williamson. 2009

239 citations recorded by the source corpus.

We study losses for binary classification and class probability estimation and extend the understanding of them from margin losses to general composite losses which are the composition of a proper loss with a link function. We characterise when margin losses can be proper composite losses, explicitly show how to determine a symmetric loss in full from half of one of its partial losses, introduce an intrinsic parametrisation of composite binary losses and give a complete characterisation of the relationship between proper losses and "classification calibrated" losses. We also consider the question of the "best" surrogate binary loss. We introduce a precise notion of "best" and show there exist situations where two convex surrogate losses are incommensurable. We provide a complete explicit characterisation of the convexity of composite binary losses in terms of the link function and the weight function associated with the proper loss which make up the composite loss. This characterisation suggests new ways of "surrogate tuning" as well as providing an explicit characterisation of when Bregman divergences on the unit interval are convex in their second argument. Finally, in an appendix we present some new algorithm-independent results on the relationship between properness, convexity and robustness to misclassification noise for binary losses and show that all convex proper losses are non-robust to misclassification noise.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Learning Optimized Risk Scores (2016, background): “ of the model class. There are some theoretical results to motivate why minimizers of the logistic loss have good AUC and risk calibration. In particular, the logistic loss is a strictly proper loss (Reid and Williamson, 2010) which yields calibrated estimates of predicted risk under the parametric ”
Entity Resolution and Federated Learning get a Federated Resolution (2018, background): “With the exception of the unhinged loss [27], this would be the case for most proper, classification calibrated and/or nonconvex losses [3, 21, 24].”
Deep learning with noisy labels: exploring techniques and remedies in medical image analysis (2019, methodology): “correcting the model predictions and only applies to composite proper loss functions ([140]), which include cross-entropy.”
Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach (2016, background): “When composite losses are also proper [33], their minimizer assumes the particular shape of the link function applied to the class-conditional probabilities p ( y | x ) : Cross-entropy and square are examples of proper composite losses.”
Optimized Risk Scores (2017, background): “Specically, the logistic loss is strictly proper, meaning it yields calibrated estimates of predicted risk under the parametric assumption that the true risk can be modeled with the logistic function [26].”
Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms (2018, background): “This form of a loss, S Ω ( ψ − 1 ( θ ); y ), is some-times called composite (Buja et al., 2005; Reid and Williamson, 2010; Williamson et al., 2016).”
Composite Multiclass Losses (2011, methodology): “They have been studied in detail when n = 2 (the “binary case”) where there is a nice integral representation [1, 2, 3], and characterization [4] when differentiable.”
Calibrated asymmetric surrogate losses (2012, background): “Proper losses, and their relationship to calibrated losses, have recently been studied by Reid and Williamson [24] and Masnadi-Shirazi and Vasconcelos [17].”
A Hybrid Loss for Multiclass and Structured Prediction (2014, methodology): “For an analysis of Fisher consistency for density estimation, we refer the reader to [15].”
Learning with Symmetric Label Noise: The Importance of Being Unhinged (2015, background): “[2009], Reid and Williamson [2011] in the context of maximum mean discrepancy; see the Appendix.”
On Symmetric Losses for Learning from Corrupted Labels (2019, background): “For example, we can recover a class probability η ( x ) from a prediction function if a loss ‘ is a proper composite loss [Buja et al., 2005; Reid & Williamson, 2010].”
Surrogate regret bounds for bipartite ranking via strongly proper losses (2012, methodology): “Our proof technique is considerably simpler than that of [19], and relies on properties of proper (composite) losses as elucidated recently for example in [22, 23, 15, 6].”
Optimal classifier for imbalanced data using Matthews Correlation Coefficient metric (2017, unknown): “The optimal classifier defined in Theorem 1 is consistent if the estimate ^ Z x is obtained using a proper loss function [22, 23].”
Loss factorization, weakly supervised learning and label noise robustness (2016, methodology): “Notice how this does not exclude losses that are not smooth, convex, or proper [Reid and Williamson, 2010].”
Contrastive Pessimistic Likelihood Estimation for Semi-Supervised Classification (2015, background): “It is a glimpse of the earlier mentioned difficult interrelationship two different performance criteria can display [73], [76], [77], [78], which we alluded to earlier on in Section 5.”
Generative Adversarial Nets from a Density Ratio Estimation Perspective (2016, methodology): “A similar idea has also been used for GAN learning [37] and class probability estimation [35].”
On the Consistency of Ordinal Regression Methods (2014, result): “ency of binary and multiclass classication for the zero-one loss has been studied for a variety of surrogate loss functions, see e.g. (Bartlett et al., 2003; Zhang, 2004a; Tewari and Bartlett, 2007; Reid and Williamson, 2010). Some of the results in this paper generalize known results for binary cla”