Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks

By Agustinus Kristiadi, Matthias Hein, Philipp Hennig. 2020

365 citations recorded by the source corpus.

The point estimates of ReLU classification networks---arguably the most widely used neural network architecture---have been shown to yield arbitrarily high confidence far away from the training data. This architecture, in conjunction with a maximum a posteriori estimation scheme, is thus not calibrated nor robust. Approximate Bayesian inference has been empirically demonstrated to improve predictive uncertainty in neural networks, although the theoretical analysis of such Bayesian approximations is limited. We theoretically analyze approximate Gaussian distributions on the weights of ReLU networks and show that they fix the overconfidence problem. Furthermore, we show that even a simplistic, thus cheap, Bayesian approximation, also fixes these issues. This indicates that a sufficient condition for a calibrated uncertainty on a ReLU network is "to be a bit Bayesian". These theoretical results validate the usage of last-layer Bayesian approximation and motivate a range of a fidelity-cost trade-off. We further validate these findings empirically via various standard experiments using common deep ReLU networks and Laplace approximations.

Subfields: Artificial Intelligence, Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, methodology): “…et al., 2018; Shu et al., 2017), modifying the output layer (Bendale and Boult, 2016; Tagasovska and Lopez-Paz, 2019; Calandra et al., 2016; Macêdo et al., 2020), or computing a closed-form posterior for the last dense layer only (Riquelme et al., 2018; Snoek et al., 2015; Kristiadi et al., 2020).”
Uncertainty Estimation with Infinitesimal Jackknife, Its Distribution and Mean-Field Approximation (2020, background): “To this end, many approaches assume layer-wise independence [41] and low-rank components [33, 38] and in some cases, restrict uncertainty quantiﬁcation to only a few layers [48] – in particular, the last layer [26].”
Hands-On Bayesian Neural Networks—A Tutorial for Deep Learning Users (2020, methodology): “hitectures that are used today in deep learning. Recent research has also shown that being only approximately Bayesian is enough to get a decent, correctly calibrated model with uncertainty estimates [46]. In this section, we present how Bayesian methods can be adapted specifically or approximated f”
Laplace Redux - Effortless Bayesian Deep Learning (2021, methodology): “This last-layer LA is cost-effective yet compelling both theoretically and in practice [28].”
A survey of uncertainty in deep neural networks (2021, methodology): “Following several works based on deep ensembles [31], [275] and BNNs, [272], [209], [204] also compared their performance to other methods based on the resulting calibration.”
Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting (2020, background): “, 2020) and fix overconfidence problems (Kristiadi et al., 2020) Another related line of research is Randomized Smoothing (Duchi et al.”