To Trust Or Not To Trust A Classifier

By Heinrich Jiang, Been Kim, Maya R. Gupta. 2018

524 citations recorded by the source corpus.

Knowing when a classifier's prediction can be trusted is useful in many applications and critical for safely using AI. While the bulk of the effort in machine learning research has been towards improving classifier performance, understanding when a classifier's predictions should and should not be trusted has received far less attention. The standard approach is to use the classifier's discriminant or confidence score; however, we show there exists an alternative that is more effective in many situations. We propose a new score, called the trust score, which measures the agreement between the classifier and a modified nearest-neighbor classifier on the testing example. We show empirically that high (low) trust scores produce surprisingly high precision at identifying correctly (incorrectly) classified examples, consistently outperforming the classifier's confidence score as well as many other baselines. Further, under some mild distributional assumptions, we show that if the trust score for an example is high (low), the classifier will likely agree (disagree) with the Bayes-optimal classifier. Our guarantees consist of non-asymptotic rates of statistical consistency under various nonparametric settings and build on recent developments in topological data analysis.

Subfields: Computational Theory and Mathematics, Statistics and Probability, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Statistics and Probability

In Safety, Robustness & Interpretability

Citations in the corpus

The Right Tool for the Job: Matching Model and Instance Complexities (2020, background): “Calibration Classifiers’ confidence scores are not always reliable (Jiang et al., 2018).”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, background): “Recent work addresses this requirement either through saliency-map visualizations [61, 77, 49], attention mechanism [75, 79, 51, 14], or other analysis [30, 39, 56, 58].”
The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations (2019, neutral): “In a similar fashion, [ Jiang et al. , 2018 ] try to identify which predictions can be trusted in regard to ground truth instances based on their distance to training data.”
Deep Weighted Averaging Classifiers (2018, methodology): “Jiang et al. [17] present a method for determining how much any existing classifier should be trusted on any given test point, based on the relative distance to the most-likely and second-most-likely clusters, with clusters based on a pruned training set.”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, methodology): “[2] presented a novel score dubbed as trust score that estimated the agreement between a modified nearest-neighbor classifier and classifier on the testing example.”
Consistent Estimators for Learning to Defer to an Expert (2020, background): “, 2019), a host of different methods exist for estimating a classifier’s confidence on new examples including trust scores (Jiang et al., 2018), Monte-Carlo dropout for neural networks (Gal & Ghahramani, 2016) among many others.”
A Unifying Review of Deep and Shallow Anomaly Detection (2020, background): “In most applications, however, accuracy alone is not sufficient [334], [516], and further criteria, such as interpretability [249], [517] and trustworthiness [475], [518], [519], are equally critical, as demonstrated in Sections VII and VIII.”
Interpretable Counterfactual Explanations Guided by Prototypes (2019, methodology): “This approach is similar to (Jiang et al. 2018) who use class speciﬁc k-d trees to measure the agreement between a classiﬁer and a modiﬁed nearest neighbour classiﬁer on test instances.”
SSD: A Unified Framework for Self-Supervised Outlier Detection (2021, methodology): “Supervised detectors have been most successful with complex input modalities, such as images and language (Chalapathy et al., 2018a; DeVries & Taylor, 2018; Dhamija et al., 2018; Jiang et al., 2018; Yoshihashi et al., 2018; Lee et al., 2018a).”
Anomalous Instance Detection in Deep Learning: A Survey (2020, methodology): “In [21], a trust score is proposed to know whether the prediction of a test example by a classifier can be trusted.”
Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models (2023, methodology): “Specifically, the confidence scores are generally associated with both the input and the prediction and can be expressed as C(x, y) [4, 6, 17, 28].”
Attention-Based Prototypical Learning Towards Interpretable, Confident and Robust Deep Neural Networks (2019, methodology): “In (Jiang et al., 2018), a confidence metric is proposed based on the agreement between the classifier and”
Energy-Based Open-World Uncertainty Modeling for Confidence Calibration (2021, methodology): “Baselines We compare our approach with nine base-line methods: MC-Dropout [7], Temperature Scaling [14], Mixup [37], Label Smoothing [36], TrustScore [20], JEM [12], DBLE [40], and OvA DM [34].”
To Trust or to Think (2021, background): “Thus, numerous studies have highlighted the importance of calibrated trust in improving AI-assisted decision-making [7, 9, 31, 44, 51, 53, 67].”
Uncertainty Sets for Image Classifiers using Conformal Prediction (2020, methodology): “This approach incurs significant computational expense and implementational complexity, which more recent approaches address with model ensembles (Lakshminarayanan et al., 2017; Jiang et al., 2018) or approximations of Bayesian inference for the parameters of the last layer only (Riquelme et al.”
BERT Loses Patience: Fast and Robust Inference with Early Exit (2020, methodology): “As revealed by prior work [32, 33], prediction of probability distributions (i.”
Interpreting Neural Networks with Nearest Neighbors (2018, background): “Recent work decides when to trust a neural network model (Ribeiro et al., 2016; Doshi-Velez and Kim, 2017; Jiang et al., 2018).”