Revisiting the Calibration of Modern Neural Networks
By Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Ann Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, Mario Lucic. 2021
562 citations recorded by the source corpus.
Accurate estimation of predictive uncertainty (model calibration) is essential for the safe application of neural networks. Many instances of miscalibration in modern neural networks have been reported, suggesting a trend that newer, more accurate models produce poorly calibrated predictions. Here, we revisit this question for recent state-of-the-art image classification models. We systematically relate model calibration and accuracy, and find that the most recent models, notably those not using convolutions, are among the best calibrated. Trends observed in prior model generations, such as decay of calibration with distribution shift or model size, are less pronounced in recent architectures. We also show that model size and amount of pretraining do not fully explain these differences, suggesting that architecture is a major determinant of calibration properties.
Subfields: Artificial Intelligence, Evaluation & Benchmarks, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Neil Houlsby
- Quoc V. Le
- Jianfeng Gao
- Mohit Bansal
- Yoshua Bengio
- Percy Liang
- Xiaohua Zhai
- Mostafa Dehghani
In Evaluation & Benchmarks
- Neil Houlsby
- Xiaohua Zhai
- Lucas Beyer
- Alexander Kolesnikov
- Sjoerd van Steenkiste
- Mitchell Wortsman
- Gabriel Ilharco
- Fisher Yu
In Safety, Robustness & Interpretability
- Justin Gilmer
- Ludwig Schmidt
- Michael Tschannen
- Xiao Wang
- Yi Tay
- Alexander Kolesnikov
- Carlos Riquelme
- Yelong Shen
Citations in the corpus
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (2023, background): “Calibration seeks to address the issue by aligning the model’s confidence with the accuracy of samples within the same confidence level (Guo et al., 2017; Minderer et al., 2021).”
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “Early work on probabilistic uncertainty estimation in NLP primarily focuses on classification (Guo et al., 2017; Minderer et al., 2021) and text regression (Glushkova et al.”
- Teaching Models to Express Their Uncertainty in Words (2022, background): “Prior work on calibration focuses primarily on the classification setting, where models output a probability distribution over the set of possible classes (Guo et al., 2017; Mukhoti et al., 2020; Minderer et al., 2021), corresponding to what we call the “answer logit”.”
- Scaling Vision Transformers to 22 Billion Parameters (2023, methodology): “The conclusion holds both without (left) and with (right) a temperature-scaling of the logits that was observed to better capture the calibration trends across model families (Minderer et al., 2021).”
- How Do Vision Transformers Work? (2022, background): “4 for a more detailed analysis. and ViTs predict well-calibrated uncertainty (Minderer et al., 2021).”
- CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration (2022, methodology): “Based on the observation that CLIP models are well calibrated [44] (i.e., CLIP confidence is indicative of CLIP predictive accuracy), a high patch probability should correlate with the object being object localization saliency free space path frontier occupied (left) Back-projected Grad-CAM [61]…”