Convex Calibration Dimension for Multiclass Loss Matrices

By Harish Guruprasad Ramaswamy, Shivani Agarwal. 2014

48 citations recorded by the source corpus.

We study consistency properties of surrogate loss functions for general multiclass learning problems, defined by a general multiclass loss matrix. We extend the notion of classification calibration, which has been studied for binary and multiclass 0-1 classification problems (and for certain other specific learning problems), to the general multiclass setting, and derive necessary and sufficient conditions for a surrogate loss to be calibrated with respect to a loss matrix in this setting. We then introduce the notion of convex calibration dimension of a multiclass loss matrix, which measures the smallest 'size' of a prediction space in which it is possible to design a convex surrogate that is calibrated with respect to the loss matrix. We derive both upper and lower bounds on this quantity, and use these results to analyze various loss matrices. In particular, we apply our framework to study various subset ranking losses, and use the convex calibration dimension as a tool to show both the existence and non-existence of various types of convex calibrated surrogates for these losses. Our results strengthen recent results of Duchi et al. (2010) and Calauzenes et al. (2012) on the non-existence of certain types of convex calibrated surrogates in subset ranking. We anticipate the convex calibration dimension may prove to be a useful tool in the study and design of surrogate losses for general multiclass learning problems.

Subfields: Artificial Intelligence, Computational Mechanics, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Evaluation & Benchmarks

Citations in the corpus

Quantifying Learning Guarantees for Convex but Inconsistent Surrogates (2018, methodology): “Consistency of convex surrogates has been the central question of many studies for such problems as binary classification [2, 24, 19], multi-class classification [23, 21, 1, 17], ranking [11, 4, 5, 18, 15] and, more recently, structured prediction [7, 14].”
Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings (2015, methodology): “…et al. (2012, 2013), Scott (2012), Duchi et al. (2013), Menon and Williamson (2014), Agarwal and Agarwal (2015), Frongillo and Kash (2015) and Ramaswamy and Agarwal (2016) and the many more reference mentioned therein), I would like to use this contribution to describe briefly the role of…”
Cross-Entropy Loss Functions: Theoretical Analysis and Applications (2023, unknown): “…2007; Dogan et al., 2016; Wang & Scott, 2020; Ramaswamy & Agarwal, 2012; Narasimhan et al., 2015; Agarwal & Agarwal, 2015; Williamson et al., 2016; Ramaswamy & Agarwal, 2016; Finocchiaro et al., 2019; Frongillo & Waggoner, 2021; Finocchiaro et al., 2022; Wang & Scott, 2023), multi-label…”