Learning for Single-Shot Confidence Calibration in Deep Neural Networks Through Stochastic Inferences
By Seonguk Seo, Paul Hongsuck Seo, Bohyung Han. 2018
81 citations recorded by the source corpus.
We propose a generic framework to calibrate accuracy and confidence of a prediction in deep neural networks through stochastic inferences. We interpret stochastic regularization using a Bayesian model, and analyze the relation between predictive uncertainty of networks and variance of the prediction scores obtained by stochastic inferences for a single example. Our empirical study shows that the accuracy and the score of a prediction are highly correlated with the variance of multiple stochastic inferences given by stochastic depth or dropout. Motivated by this observation, we design a novel variance-weighted confidence-integrated loss function that is composed of two cross-entropy loss terms with respect to ground-truth and uniform distribution, which are balanced by variance of stochastic prediction scores. The proposed loss function enables us to learn deep neural networks that predict confidence calibrated scores using a single inference. Our algorithm presents outstanding confidence calibration performance and improves classification accuracy when combined with two popular stochastic regularization techniques---stochastic depth and dropout---in multiple models and datasets; it alleviates overconfidence issue in deep neural networks significantly by training networks to achieve prediction accuracy proportional to confidence of prediction.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dustin Tran
- Michael W. Dusenberry
- Ghassen Jerfel
- Jeremy Nixon
- Muhammad Shahzad
- Cedrique Rovile Njieutcheu Tassi
- Richard Bamler
- Matthias Humt
Citations in the corpus
- Measuring Calibration in Deep Learning (2019, background): “A suite of recent papers (e.g., Lee et al., 2017; Vaicenavicius et al., 2019; Thulasidasan et al., 2019; Kumar et al., 2018; Guo et al., 2017; Seo et al., 2019) use ECE to validate their models’ calibration.”
- A survey of uncertainty in deep neural networks (2021, background): “[266] extended the idea of label smoothing and directly aimed at reducing the model uncertainty.”
- Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation (2019, background): “[22] proposed a variance-weighted loss function that enables learning single-shot calibration scores.”
- Multivariate Confidence Calibration for Object Detection (2020, methodology): “In other words, we do not modify the learning objective of a detector and thus our work is orthogonal to ongoing research in regularization methods for calibration [15, 22, 18].”