Confusion Matrix Stability Bounds for Multiclass Classification

By Pierre Machart, Liva Ralaivola. 2012

31 citations recorded by the source corpus.

In this paper, we provide new theoretical results on the generalization properties of learning algorithms for multiclass classification problems. The originality of our work is that we propose to use the confusion matrix of a classifier as a measure of its quality; our contribution is in the line of work which attempts to set up and study the statistical properties of new evaluation measures such as, e.g. ROC curves. In the confusion-based learning framework we propose, we claim that a targetted objective is to minimize the size of the confusion matrix C, measured through its operator norm ||C||. We derive generalization bounds on the (size of the) confusion matrix in an extended framework of uniform stability, adapted to the case of matrix valued loss. Pivotal to our study is a very recent matrix concentration inequality that generalizes McDiarmid's inequality. As an illustration of the relevance of our theoretical results, we show how two SVM learning procedures can be proved to be confusion-friendly. To the best of our knowledge, the present paper is the first that focuses on the confusion matrix from a theoretical point of view.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Benefits of Intermediate Annotations in Reading Comprehension (2020, methodology): “We compute confusion loss (Machart and Ralaivola, 2012) as the metric for this, which measures error diffusion by computing the highest singular value of the unnormalized confusion matrix after setting the diagonal elements (i.e. true positives), to zero (Koc¸o and Capponi, 2013) (lower confusion…”
Consistent Multiclass Algorithms for Complex Performance Measures (2015, background): “…matrix include the macro F 1 -measure (Lewis, 1991), the H-mean (Kennedy et al., 2009), the Q-mean (Lawrence et al., 1998), the spectral norm measure (Ralaivola, 2012; Machart & Ralaivola, 2012; Koco & Capponi, 2013), and the min-max measure in detection theory (Vincent, 1994); see Table 1.”