Estimating the accuracies of multiple classifiers without labeled data

By Ariel Jaffe, Boaz Nadler, Yuval Kluger. 2014

74 citations recorded by the source corpus.

In various situations one is given only the predictions of multiple classifiers over a large unlabeled test data. This scenario raises the following questions: Without any labeled data and without any a-priori knowledge about the reliability of these different classifiers, is it possible to consistently and computationally efficiently estimate their accuracies? Furthermore, also in a completely unsupervised manner, can one construct a more accurate unsupervised ensemble classifier? In this paper, focusing on the binary case, we present simple, computationally efficient algorithms to solve these questions. Furthermore, under standard classifier independence assumptions, we prove our methods are consistent and study their asymptotic error. Our approach is spectral, based on the fact that the off-diagonal entries of the classifiers' covariance matrix and 3-d tensor are rank-one. We illustrate the competitive performance of our algorithms via extensive experiments on both artificial and real datasets.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Unsupervised Risk Estimation with only Structural Assumptions (2016, methodology): “…et al. (2011) assume Gaussianity of model scores, while a line of work starting with Dawid & Skene (1979) assume multiple classifiers with known dependency structure, which specifies a complete generative distribution for the 0/1-loss (Zhang et al., 2014; Platanios, 2015; Jaffe et al., 2015).”