Why ReLU Networks Yield High-Confidence Predictions Far Away From the Training Data and How to Mitigate the Problem

By Matthias Hein, Maksym Andriushchenko, Julian Bitterwolf. 2018

630 citations recorded by the source corpus.

Classifiers used in the wild, in particular for safety-critical systems, should not only have good generalization properties but also should know when they don't know, in particular make low confidence predictions far away from the training data. We show that ReLU type neural networks which yield a piecewise linear classifier function fail in this regard as they produce almost always high confidence predictions far away from the training data. For bounded domains like images we propose a new robust optimization technique similar to adversarial training which enforces low confidence predictions far away from the training data. We show that this technique is surprisingly effective in reducing the confidence of predictions far away from the training data while maintaining high confidence predictions and test error on the original classification task compared to standard training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Shortcut learning in deep neural networks (2020, background): “In most cases, however, it is hard to understand the implicit priors in a DNN and even standard elements like ReLU activations can lead to unexpected effects like unwarranted confidence [109].”
Dynamic Neural Networks: A Survey (2021, background): “Note that the overconfidence issue in deep models [201], [202] might affect the effective-”
Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, background): “These are achieved by either replacing the loss function (Hein et al., 2019; Malinin and Gales, 2018a;b; Sensoy et al., 2018; Shu et al., 2017), modifying the output layer (Bendale and Boult, 2016; Tagasovska and Lopez-Paz, 2019; Calandra et al.”
Posterior Meta-Replay for Continual Learning (2021, background): “Thus, the behaviour of aleatoric uncertainty as reﬂected in p ( Y | W ; X ) for OOD data (e.g., outside the support of p ( X ) ) is not calibrated and can indeed be harmful for OOD detection (e.g., [20]).”
Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks (2020, methodology): “Lemma A.2 (Hein et al., 2019).”
Towards neural networks that provably know when they don't know (2019, methodology): “For a ReLU network the classifier function f is componentwise a continuous piecewise affine function and has been shown to produce asymptotically arbitrarily highly confident predictions (Hein et al., 2019), i.”
Learning Placeholders for Open-Set Recognition (2021, background): “However, deep learning methods tend to overﬁt the training instances and produce overconﬁdent predictions [27, 8, 9].”
Energy-based Out-of-distribution Detection (2020, background): “A theoretical investigation [13] shows that neural networks with ReLU activation can produce arbitrarily high softmax conﬁdence for OOD inputs.”
Generating Accurate Pseudo-Labels in Semi-Supervised Learning and Avoiding Overconfident Predictions via Hermite Polynomial Activations (2019, background): “In other words, [27] and [12] taken together hint at the possibility that ReLUs may, at least, partly contribute to the confirmation bias in PL based SSL methods.”
Isotropic Maximization Loss and Entropic Score: Fast, Accurate, Scalable, Unexposed, Turnkey, and Native Neural Networks Out-of-Distribution Detection. (2019, neutral): “Finally, our solution does not rely on adversarial training as the ACET proposed in Hein et al. (2018). Therefore, training a neural network with IsoMax is as fast as training with SoftMax.”
How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks (2020, background): “MLPs have ﬁnitely many linear regions (Arora et al., 2018; Hanin & Rolnick, 2019; Hein et al., 2019).”
MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space (2021, background): “A plethora of literature has emerged in addressing the problem of OOD detection [3, 14, 16, 24, 27, 29, 35, 4, 20, 31].”
Generalized Out-of-Distribution Detection: A Survey (2021, neutral): “…(Wang et al., 2021), ensembling with leaving-out strategy (Vyas et al., 2018), adversarial training (Bitterwolf et al., 2020; Chen et al., 2020b; Hein et al., 2019; Choi & Chung, 2020; Chen et al., 2021c), stronger data augmentation (Hein et al., 2019; Thulasidasan et al., 2019; Yun et al.,…”
Generative-Discriminative Feature Representations for Open-Set Recognition (2020, background): “Since then, several other works have analyzed this challenge in the context of deep networks [22],[11].”
Non-Parametric Calibration for Classification (2019, background): “It was recently observed that many modern CNNs are overconﬁdent (Lakshminarayanan et al., 2017, Hein et al., 2019) and miscalibrated (Guo et al., 2017).”
The Overlooked Elephant of Object Detection: Open Set (2020, background): “Existing approaches: There are many approaches that attempt to solve the open-set classiﬁcation task [10, 11], but the majority of these approaches have been either restricted to smaller datasets such as MNIST or focus on identifying unknown knowns, i.e., novel views of known classes such as…”
OpenOOD: Benchmarking Generalized Out-of-Distribution Detection (2022, background): “Besides, even with the same benchmark, some technical details such as image preprocessing procedures [25, 26], are not unified, increasing the difficulty of a fair comparison.”
ReAct: Out-of-distribution Detection With Rectified Activations (2021, background): “[13] showed that neural networks with ReLU activation can lead to arbitrary high activation for inputs far away from the training data.”