On Calibration of Modern Neural Networks

By Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger. 2017

8,827 citations recorded by the source corpus.

Confidence calibration -- the problem of predicting probability estimates representative of the true correctness likelihood -- is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-of-the-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Yoshua Bengio
  3. Quoc V. Le
  4. Percy Liang
  5. Sergey Levine
  6. Jianfeng Gao
  7. Luke S. Zettlemoyer
  8. Ruslan Salakhutdinov

In Safety, Robustness & Interpretability

  1. Matthias Hein
  2. Dan Hendrycks
  3. Percy Liang
  4. Nicholas Carlini
  5. Eric Wallace
  6. Prateek Mittal
  7. Dawn Song
  8. Yixuan Li

Citations in the corpus