Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration

By Meelis Kull, Miquel Perello-Nieto, Markus Kängsepp, T. S. Filho, Hao Song, Peter A. Flach. 2019

547 citations recorded by the source corpus.

Class probabilities predicted by most multiclass classifiers are uncalibrated, often tending towards over-confidence. With neural networks, calibration can be improved by temperature scaling, a method to learn a single corrective multiplicative factor for inputs to the last softmax layer. On non-neural models the existing methods apply binary calibration in a pairwise or one-vs-rest fashion. We propose a natively multiclass calibration method applicable to classifiers from any model class, derived from Dirichlet distributions and generalising the beta calibration method from binary classification. It is easily implemented with neural nets since it is equivalent to log-transforming the uncalibrated probabilities, followed by one linear layer and softmax. Experiments demonstrate improved probabilistic predictions according to multiple measures (confidence-ECE, classwise-ECE, log-loss, Brier score) across a wide range of datasets and classifiers. Parameters of the learned Dirichlet calibration map provide insights to the biases in the uncalibrated model.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Dark Experience for General Continual Learning: a Strong, Simple Baseline (2020, background): “Recent works find out that modern Deep Networks – despite largely outperforming the ones from a decade ago – are less calibrated [13], as they tend to yield overconfident predictions [22].”
Verified Uncertainty Calibration (2019, methodology): “Prior works [9, 15, 17] propose methods for multiclass calibration but only measure top-label calibration—[24] and concurrent work to ours [26] define similar per-class calibration metrics where temperature scaling [9] is worse than vector scaling despite having better top-label calibration.”
Local Temperature Scaling for Probability Calibration (2020, methodology): “[28] Kull, M.”
Unsolved Problems in ML Safety (2021, background): “…express their domain of competence in a more representative and meaningful way, researchers could further improve model calibration on typical testing data [68, 133, 113, 109, 205, 107, 108, 124], though the greater challenge is calibration on testing data that is unlike the training data [137].”
A survey of uncertainty in deep neural networks (2021, methodology): “[284] were motivated by non-neural network calibration methods, where the calibration is performed classwise as a one-vs-all binary calibration.”