Uncertainty Sets for Image Classifiers using Conformal Prediction

By Anastasios Nikolas Angelopoulos, Stephen Bates, Jitendra Malik, Michael I. Jordan. International Conference on Learning Representations, 2020

488 citations recorded by the source corpus.

Convolutional image classifiers can achieve high predictive accuracy, but quantifying their uncertainty remains an unresolved challenge, hindering their deployment in consequential settings. Existing uncertainty quantification techniques, such as Platt scaling, attempt to calibrate the network's probability estimates, but they do not have formal guarantees. We present an algorithm that modifies any classifier to output a predictive set containing the true label with a user-specified probability, such as 90%. The algorithm is simple and fast like Platt scaling, but provides a formal finite-sample coverage guarantee for every model and dataset. Furthermore, our method generates much smaller predictive sets than alternative methods, since we introduce a regularizer to stabilize the small scores of unlikely classes after Platt scaling. In experiments on both Imagenet and Imagenet-V2 with a ResNet-152 and other classifiers, our scheme outperforms existing approaches, achieving exact coverage with sets that are often factors of 5 to 10 smaller.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/9cd7c6d19dedc5395155492e0193f399bbabec49

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Few-shot Conformal Prediction with Auxiliary Tasks (2021, background): “Accurate estimates of uncertainty are important for difficult or sensitive prediction problems that have variable accuracy (Amodei et al., 2016; Jiang et al., 2012; 2018; Angelopoulos et al., 2021).”
Conformal prediction with conditional guarantees (2023, unknown): “This procedure is known as temperature scaling and it has been found to increase the accuracy of probabilities output by neural networks (Angelopoulos et al. (2021), Guo et al. (2017)).”
Conformal Risk Control (2022, background): “Recently there have been many extensions of the conformal algorithm, mainly targeting deviations from exchangeability (Tibshirani et al., 2019; Gibbs & Candes, 2021; Barber et al., 2022; Fannjiang et al., 2022) and improved conditional coverage (Barber et al., 2020; Romano et al., 2019; Guan, 2020;…”
Distribution-Free, Risk-Controlling Prediction Sets (2021, background): “…prediction is a generic approach, and much recent work has focused on designing speciﬁc conformal procedures to have good performance according to metrics such as small set sizes [14], approximate coverage in all regions of feature space [15–21], and errors balanced across classes [14, 22–24].”
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners (2023, background): “However, the resulting prediction sets are not guaranteed to achieve 1 − ϵ coverage as the probabilities can be miscali-brated [15], and often include additional unnecessary options [16].”