Calibration tests in multi-class classification: A unifying framework

By David Widmann, Fredrik Lindsten, Dave Zachariah. 2019

120 citations recorded by the source corpus.

In safety-critical applications a probabilistic model is usually required to be calibrated, i.e., to capture the uncertainty of its predictions accurately. In multi-class classification, calibration of the most confident predictions only is often not sufficient. We propose and study calibration measures for multi-class classification that generalize existing measures such as the expected calibration error, the maximum calibration error, and the maximum mean calibration error. We propose and evaluate empirically different consistent and unbiased estimators for a specific class of measures based on matrix-valued kernels. Importantly, these estimators can be interpreted as test statistics associated with well-defined bounds and approximations of the p-value under the null hypothesis that the model is calibrated, significantly improving the interpretability of calibration measures, which otherwise lack any meaningful unit or scale.

Subfields: Artificial Intelligence, Control and Systems Engineering, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Percy Liang
2. Philip H. S. Torr
3. Tengyu Ma
4. Quoc V. Le
5. Bernt Schiele
6. Yoshua Bengio
7. Mohit Bansal
8. Luke S. Zettlemoyer

In Control and Systems Engineering

1. Philip H. S. Torr
2. Michael C. Mozer
3. Percy Liang
4. Jonathon Shlens
5. Tengyu Ma
6. Ananya Kumar

In Statistics and Probability

1. Tengyu Ma
2. Philip H. S. Torr
3. Michael C. Mozer
4. Puneet Kumar Dokania
5. Percy Liang
6. Jonathon Shlens
7. Ananya Kumar
8. Nicholas Cain

Citations in the corpus

• Verified Uncertainty Calibration (2019, background): “Concurrent work to ours [39] also notice that using the plugin calibration error estimator to test for calibration leads to rejecting well-calibrated models too often.”
• Mitigating bias in calibration error estimation (2020, background): “For example, Widmann et al. (2019) proposed a kernel-based calibration error, KCE, which has no explicit dependence on the model’s calibration function.”
• Calibrating Deep Neural Networks using Focal Loss (2020, background): “A stronger definition of calibration would require the probabilities of all the classes in the softmax distribution to be calibrated [14, 36, 39, 15].”