Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
By Balaji Lakshminarayanan, Alexander Pritzel, Charles Blundell. 2016
8,039 citations recorded by the source corpus.
Deep neural networks (NNs) are powerful black box predictors that have recently achieved impressive performance on a wide spectrum of tasks. Quantifying predictive uncertainty in NNs is a challenging and yet unsolved problem. Bayesian NNs, which learn a distribution over weights, are currently the state-of-the-art for estimating predictive uncertainty; however these require significant modifications to the training procedure and are computationally expensive compared to standard (non-Bayesian) NNs. We propose an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates. Through a series of experiments on classification and regression benchmarks, we demonstrate that our method produces well-calibrated uncertainty estimates which are as good or better than approximate Bayesian NNs. To assess robustness to dataset shift, we evaluate the predictive uncertainty on test examples from known and unknown distributions, and show that our method is able to express higher uncertainty on out-of-distribution examples. We demonstrate the scalability of our method by evaluating predictive uncertainty estimates on ImageNet.
Subfields: Artificial Intelligence, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Mohit Bansal
- Percy Liang
- Quoc V. Le
- Jianfeng Gao
- Ruslan Salakhutdinov
- Luke S. Zettlemoyer
In Evaluation & Benchmarks
- Yejin Choi
- Mohit Bansal
- Percy Liang
- Hannaneh Hajishirzi
- Yoshua Bengio
- Sergey Levine
- Robin Jia
- Noah A. Smith
Citations in the corpus
- The Power of Scale for Parameter-Efficient Prompt Tuning (2021, background): “Ensembles of neural models trained from different initializations on the same data are widely observed to improve task performance (Hansen and Salamon, 1990) and are useful for estimating model uncertainty (Lakshminarayanan et al., 2017).”
- Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, methodology): “The experimental protocol we use to probe underspecification is closely related to uncertainty quantification approaches based on deep ensembles (e.g., Lakshminarayanan et al., 2017; Dusenberry et al., 2020).”
- MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, methodology): “For this reason, it is often used as the unlabeled data loss in SSL [25, 44] as well as a measure of predictive uncertainty [26].”
- Scalable agent alignment via reward modeling: a research direction (2018, background): “So far model ensembles provide a very strong baseline (Lakshminarayanan et al., 2017).”
- Repurposing Pretrained Models for Robust Out-of-domain Few-Shot Learning (2021, background): “Lakshminarayanan et al. (2017) showed that they produce predictive uncertainty estimates comparable in quality to BNNs. Deep ensembles, however, are not directly applicable for FSL tasks, as training randomly initialized parameters from scratch with a limited amount of training data yields poor…”
- Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “Combining the outputs of many models is a foundational technique for improving the accuracy and robustness of machine learning models (Dietterich, 2000; Bauer and Kohavi, 1999; Breiman, 1996; Friedman et al., 2001; Lakshminarayanan et al., 2017; Freund and Schapire, 1997).”
- On Calibration of Modern Neural Networks (2017, methodology): “Lakshminarayanan et al. (2016) use ensembles of networks to obtain uncertainty estimates.”
- AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty (2019, background): “liability of real-world machine learning systems. Calibration under Data Shift. Guo et al.(2017);Nguyen & O’Connor(2015) propose metrics for determining the calibration of machine learning models.Lakshminarayanan et al. (2017) find that simply ensembling classifier predictions improves prediction ”
- Unsolved Problems in ML Safety (2021, neutral): “To help models express their domain of competence in a more representative and meaningful way, researchers could further improve model calibration on typical testing data [68, 131, 112, 108, 205, 106, 107, 123], though the greater challenge is calibration on testing data that is unlike the training ”
- Evidential Deep Learning to Quantify Classification Uncertainty (2018, methodology): “1 In this section, we compared the following approaches: (a) L2 corresponds to the standard deterministic neural nets with softmax output and weight decay, (b) Dropout refers to the uncertainty estimation model used in [8], (c) Deep Ensemble refers to the model used in [21], (d) FFG refers to”
- Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, methodology): “[42] B.”
- A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, methodology): “We evaluated ensembles of independently trained SGD solutions (Deep Ensembles, [29]) on PreResNet-164 on CIFAR-100.”
- Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning (2018, methodology): “Because dropout can be seen as an ensembling method, this approach naturally generalizes to using ensembles of models as a proxy to estimate predictive uncertainty [46].”
- BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “Ensembles of neural networks, even of size three and five, have been shown in some cases to give more reliable uncertainty estimates than other leading approaches such as BNNs (Lakshminarayanan, Pritzel, and Blundell 2017; Beluch et al. 2018; Choi et al. 2016; Snoek et al. 2019; Zaidi et al. 2020).”
- Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data (2019, neutral): “ensemble methods [19].”
- Deep Factors for Forecasting (2019, methodology): “Future work includes comparing variational dropout (Gal & Ghahramani, 2016) or Deep En-semble (Lakshminarayanan et al., 2017) of non-probabilistic DNN models (e.g., RNNForecaster (cf. 5.1)) for uncertainty. using the synthetic example in Section 5.1.”
- Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “While a number of prior works have explored uncertainty-aware deep neural network models [Neal, 1995, Lakshminarayanan et al., 2017], including in the context of RL [Gal et al., 2016, Depeweg et al., 2016], our work is, to our knowledge, the first to bring these components together in a deep MBRL…”