Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

By Yaniv Ovadia, Emily Fertig, Jie Jessie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, Jasper Snoek. 2019

2,269 citations recorded by the source corpus.

Modern machine learning methods including deep learning have achieved great success in predictive accuracy for supervised learning tasks, but may still fall short in giving useful estimates of their predictive {\em uncertainty}. Quantifying uncertainty is especially critical in real-world settings, which often involve input distributions that are shifted from the training distribution due to a variety of factors including sample bias and non-stationarity. In such settings, well calibrated uncertainty estimates convey information about when a model's output should (or should not) be trusted. Many probabilistic deep learning methods, including Bayesian-and non-Bayesian methods, have been proposed in the literature for quantifying predictive uncertainty, but to our knowledge there has not previously been a rigorous large-scale empirical comparison of these methods under dataset shift. We present a large-scale benchmark of existing state-of-the-art methods on classification problems and investigate the effect of dataset shift on accuracy and calibration. We find that traditional post-hoc calibration does indeed fall short, as do several other previous methods. However, some methods that marginalize over models give surprisingly strong results across a broad spectrum of tasks.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Measuring Massive Multitask Language Understanding (2020, background): “, 2017), especially under distribution shift (Ovadia et al., 2019).”
On Robustness and Transferability of Convolutional Neural Networks (2020, background): “Instead of requiring the model to work under various dataset shifts, one can ask an alternative question: assuming that the model will be deployed in an environment significantly different from the training one, can we at least quantify the model uncertainty for each prediction? This important prope”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, background): “[68] show that ensembles exhibit high accuracy under distribution shift.”
AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty (2019, methodology): “Even methods which aim to explicitly quantify uncertainty, such as probabilistic and Bayesian neural networks, struggle under data shift, as recently demonstrated by Ovadia et al. (2019).”
Unsolved Problems in ML Safety (2021, neutral): “To help models express their domain of competence in a more representative and meaningful way, researchers could further improve model calibration on typical testing data [68, 131, 112, 108, 205, 106, 107, 123], though the greater challenge is calibration on testing data that is unlike the training ”
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, neutral): “Ensembles of neural networks, even of size three and five, have been shown in some cases to give more reliable uncertainty estimates than other leading approaches such as BNNs (Lakshminarayanan, Pritzel, and Blundell 2017; Beluch et al. 2018; Choi et al. 2016; Snoek et al. 2019; Zaidi et al. 2020).”
Posterior Meta-Replay for Continual Learning (2021, background): “A successful approach for OOD detection are deep ensembles [68], which in the simplest case correspond to several deterministic networks trained independently, and whose predictions are aggregated during inference [35].”
Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics (2020, methodology): “The catastrophic forgetting problem manifests in many sub-domains of machine learning including continual learning [6], multi-task learning and transfer [7] and even in standard supervised learning through input distribution shift [8, 9, 10, 11, 12] and data augmentation [13].”
Simple and Scalable Epistemic Uncertainty Estimation Using a Single Deep Deterministic Neural Network (2020, background): “Snoek et al. (2019) showed that Deep Ensembles consistently outperform Bayesian neural networks that were trained using variational inference.”
BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning (2020, methodology): “Ovadia et al. (2019) used sample size 128 while we found no signiﬁcant difference between sample size 128 and 8.”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “In practice, such heuristics could be challenging for complex datasets or deep neural network models [44].”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, methodology): “This is to provide an intuitive picture of the challenge of open set recognition with neural networks and to summarize and repeat the ﬁndings of the much more detailed experiments presented in numerous prior works (Mundt et al., 2022, 2019; Nalisnick et al., 2019; Ovadia et al., 2019).”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, neutral): “These relations are further supported by previous work, which showed that deep ensembles provide well-calibrated uncertainty estimates (Lakshminarayanan et al., 2017; Gustafsson et al., 2019; Snoek et al., 2019).”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “Indeed, prior work has shown that selective prediction can improve model accuracy under distribution shifts (Pimentel et al., 2014; Hendrycks and Gimpel, 2017; Liang et al., 2018; Ovadia et al., 2019; Feng et al., 2019; Kamath et al., 2020).”
Deep Ensembles: A Loss Landscape Perspective (2019, methodology): “Figure 10(b) shows accuracy and Brier score on CIFAR-10, both on the usual test set (corresponding to the intensity = 0 column) as well as on the CIFAR-10-C benchmark proposed (Hendrycks & Dietterich, 2019) which contains corrupted versions of CIFAR-10 with varying intensity values (1-5), making it ”