Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision

By Fredrik K. Gustafsson, Martin Danelljan, Thomas Bo Schön. 2019

349 citations recorded by the source corpus.

While deep neural networks have become the go-to approach in computer vision, the vast majority of these models fail to properly capture the uncertainty inherent in their predictions. Estimating this predictive uncertainty can be crucial, for example in automotive applications. In Bayesian deep learning, predictive uncertainty is commonly decomposed into the distinct types of aleatoric and epistemic uncertainty. The former can be estimated by letting a neural network output the parameters of a certain probability distribution. Epistemic uncertainty estimation is a more challenging problem, and while different scalable methods recently have emerged, no extensive comparison has been performed in a real-world setting. We therefore accept this task and propose a comprehensive evaluation framework for scalable epistemic uncertainty estimation methods in deep learning. Our proposed framework is specifically designed to test the robustness required in real-world computer vision applications. We also apply this framework to provide the first properly extensive and conclusive comparison of the two current state-of-the- art scalable methods: ensembling and MC-dropout. Our comparison demonstrates that ensembling consistently provides more reliable and practically useful uncertainty estimates. Code is available at https://github.com/fregu856/evaluating_bdl.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, positive): “These relations are further supported by previous work, which showed that deep ensembles provide well-calibrated uncertainty estimates (Laksh-minarayanan et al., 2017; Gustafsson et al., 2019; Snoek et al., 2019).”
Deep Ensembles: A Loss Landscape Perspective (2019, methodology): “(2019) and Gustafsson et al. (2019) independently benchmarked existing methods for uncertainty quantification on a variety of datasets and architectures, and observed that ensembles tend to outperform approximate Bayesian neural networks in terms of both accuracy and uncertainty, particularly under ”
Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, background): “[12] briefly mention how deep ensembles can be viewed as samples from an approximate posterior.”
Energy-Based Models for Deep Probabilistic Regression (2020, background): “detection [13, 19] and optical flow estimation [15, 22], usually aiming to not only achieve accurate predictions, but also to provide an estimate of aleatoric uncertainty [24, 17].”
Neural Ensemble Search for Performant and Calibrated Predictions (2020, background): “[21] Fredrik K Gustafsson, Martin Danelljan, and Thomas B Schön.”
Uncertainty Estimation for Language Reward Models (2022, background): “They have been found to be a strong baseline in a variety of applications [3, 47, 30, 15], although they have been criticized on both a theoretical [28] and empirical [29] basis.”
A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving (2020, methodology): “In general, MC-Dropout provides a practical way to perform approximate inference in Bayesian Neural Netowrks (BNNs), and is scalable to large datasets and network architectures such as in image classiﬁcation [47] or semantic segmentation [42], [62].”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, background): “[422] devised a comprehensive assessment framework for scalable epistemic uncertainty estimation techniques in deep learning.”
A survey of uncertainty in deep neural networks (2021, methodology): “[56] introduced a framework for the comparison of uncertainty quantification methods with a specific focus on real life applications.”
Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning (2020, background): “These settings include, but are not limited to, regression, language modeling (Gal, 2016), image segmentation (Gustafsson et al., 2019), active learning (Settles, 2012) and reinforcement learning (Buckman et al., 2018; Chua et al., 2018).”
Hyperparameter Ensembles for Robustness and Uncertainty Quantification (2020, background): “Deep ensembles lead to remarkable predictive performance and robust uncertainty estimates [64, 28].”