Simple and Scalable Epistemic Uncertainty Estimation Using a Single Deep Deterministic Neural Network
By Joost R. van Amersfoort, Lewis Smith, Yee Whye Teh, Yarin Gal. 2020
55 citations recorded by the source corpus.
We propose a method for training a deterministic deep model that can find and reject out of distribution data points at test time with a single forward pass. Our approach, deterministic uncertainty quantification (DUQ), builds upon ideas of RBF networks. We scale training in these with a novel loss function and centroid updating scheme. By enforcing detectability of changes in the input using a gradient penalty, we are able to reliably detect out of distribution data. Our uncertainty quantification scales well to large datasets, and using a single model, we improve upon or match Deep Ensembles on notable difficult dataset pairs such as FashionMNIST vs. MNIST, and CIFAR-10 vs. SVHN, while maintaining competitive accuracy.
Subfields: Artificial Intelligence, Control and Systems Engineering, Probabilistic & Bayesian Methods
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dustin Tran
- Balaji Lakshminarayanan
- Adrian Weller
- Shreyas Padhy
- Q. Liao
- Javier Antorán
- Umang Bhatt
- Tania Bedrax-Weiss
In Control and Systems Engineering
- Abbas Khosravi
- Adrian Weller
- Mohammad Ghavamzadeh
- Paul W. Fieguth
- Saeid Nahavandi
- Balaji Lakshminarayanan
- Dustin Tran
In Probabilistic & Bayesian Methods
- Balaji Lakshminarayanan
- Javier Antorán
- Ranganath Krishnan
- Dana Rezazadegan
- Zi Lin
- Vladimir Makarenkov
- Q. Liao
- Abbas Khosravi
Citations in the corpus
- Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, methodology): “On the other hand, the lower Lipschitz bound ||h(x1)−h(x2)||H ≥ L1 ∗ ||x1−x2||X prevents the hidden representation from being unnecessarily invariant to the semantically meaningful changes in the input manifold [36, 75].”
- A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, methodology): “[209] Image Neural Networks Deterministic UQ (DUQ) L(x, y) = − ∑ c yc log(Kc)+(1−yc) log(1−Kc) ×”
- Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty (2020, methodology): “These methods can be broadly classified into two categories: Bayesian approaches [5, 20, 32, 51, 57, 63, 69, 92, 118, 184, 194] and Non-Bayesian, or frequentist, approaches [2, 98, 102, 107, 111, 177].”