Non-Parametric Calibration for Classification
By Jonathan Wenger, Hedvig Kjellström, Rudolph Triebel. 2019
95 citations recorded by the source corpus.
Many applications of classification methods not only require high accuracy but also reliable estimation of predictive uncertainty. However, while many current classification frameworks, in particular deep neural networks, achieve high accuracy, they tend to incorrectly estimate uncertainty. In this paper, we propose a method that adjusts the confidence estimates of a general classifier such that they approach the probability of classifying correctly. In contrast to existing approaches, our calibration method employs a non-parametric representation using a latent Gaussian process, and is specifically designed for multi-class classification. It can be applied to any classifier that outputs confidence estimates and is not limited to neural networks. We also provide a theoretical analysis regarding the over- and underconfidence of a classifier and its relationship to calibration, as well as an empirical outlook for calibrated active learning. In experiments we show the universally strong performance of our method across different classifiers and benchmark data sets, in particular for state-of-the art neural network architectures.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Agustinus Kristiadi
- Philipp Hennig
- Matthias Hein
- Muhammad Shahzad
- Cedrique Rovile Njieutcheu Tassi
- Marc Niethammer
- Richard Bamler
- Matthias Humt
Citations in the corpus
- Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks (2020, methodology): “In particular, the optimal temperature is found via a validation log-likelihood maximization using PyCalib (Wenger et al., 2020).”
- Local Temperature Scaling for Probability Calibration (2020, background): “Gaussian Process (GP) approaches [65] can inherently provide good uncertainty estimates, but may suffer from lower accuracy and higher computational complexity on high-dimensional classification tasks.”
- A survey of uncertainty in deep neural networks (2021, methodology): “proposed [15], [47], [48] to obtain reliable (well-calibrated)”