Lightweight Probabilistic Deep Networks

By Jochen Gast, Stefan Roth. 2018

204 citations recorded by the source corpus.

Even though probabilistic treatments of neural networks have a long history, they have not found widespread use in practice. Sampling approaches are often too slow already for simple networks. The size of the inputs and the depth of typical CNN architectures in computer vision only compound this problem. Uncertainty in neural networks has thus been largely ignored in practice, despite the fact that it may provide important information about the reliability of predictions and the inner workings of the network. In this paper, we introduce two lightweight approaches to making supervised learning with probabilistic deep networks practical: First, we suggest probabilistic output layers for classification and regression that require only minimal changes to existing networks. Second, we employ assumed density filtering and show that activation uncertainties can be propagated in a practical fashion through the entire network, again with minor changes. Both probabilistic networks retain the predictive power of the deterministic counterpart, but yield uncertainties that correlate well with the empirical error induced by their predictions. Moreover, the robustness to adversarial examples is significantly increased.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Probabilistic & Bayesian Methods

Citations in the corpus

A General Framework for Uncertainty Estimation in Deep Learning (2019, methodology): “We refer the reader to [5], [6], [24] for the details of the propagation formulas.”
Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks (2020, background): “, 1999; Guo et al., 2017; Liang et al., 2018) are unable to detect inputs with arbitrarily high confidence far away from the training data (Hein et al., 2019). Many works on uncertainty quantification in deep learning have recently been proposed. Gast & Roth (2018) proposed lightweight probabilistic”
A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving (2020, methodology): “As for the regression probability, Gaussian distributions [35], [40] or Gaussian Mixture Models (GMM) [15], [65] are often used.”
Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision (2019, methodology): “For classiﬁcation tasks, a predictive categorical distribution is commonly realized by a softmax output layer, although recent work has also explored Dirichlet models [14, 41, 34].”
ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformer (2022, methodology): “As an elegant framework for uncertainty prediction, some works [15,42,43,44,45] propose to use probabilistic model to jointly explain both flows estimations and their confidence.”
Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximation (2019, methodology): “This problem has been previously studied in the scope of uncertainty propagation in DNNs (Abdelaziz et al., 2015; Gast & Roth, 2018; Thiagarajan et al., 2018), where the goal is to analyze how the uncertainty of the input data or of the network parameters propagates through the linear and…”
Uncertainty-Aware Deep Classifiers Using Generative Models (2020, methodology): “Our work can be considered as an extension of approaches for classiﬁcation, which take the output of a neural network for an input sample to estimate parameters of a Dirichlet distribution (Sensoy, Kaplan, and Kandemir 2018; Malinin and Gales 2018; Gast and Roth 2018) for its classiﬁcation.”
Deep Evidential Regression (2019, unknown): “However, these works either rely on regularizing divergence to a fixed, well-defined prior [42, 46], require OOD training data [32, 31, 7, 19], or can only estimate aleatoric uncertainty by performing density estimation [11, 18].”
SalsaNext: Fast, Uncertainty-Aware Semantic Segmentation of LiDAR Point Clouds (2020, unknown): “r’s noise. In this case, the aleatoric uncertainty can be computed by propagating the noise through the network via Assumed Density Filtering (ADF). This approach was initially applied by Gast et al. [36], where the network’s activation functions including input and output were replaced by probabili”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, methodology): “However, as Gast and Roth [410] stated although probabilistic approaches have widely been used for several years; however, probabilistic approaches have not been comprehensively applied in practice since sampling techniques are frequently too slow.”
Energy-Based Models for Deep Probabilistic Regression (2020, methodology): “To allow for straightforward implementation and training, many of these probabilistic regression approaches however restrict the parametric model to unimodal distributions such as Gaussian [26, 7] or Laplace [24, 15, 22], still severely limiting the expressiveness of the learned conditional target d”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, background): “Gast and Roth [113] explore the possibility of replacing the deterministic outputs by probabilistic output layers.”
A Survey on Bayesian Deep Learning (2016, methodology): “Following NPN, a light-weight version [26] was proposed to speed up the training and inference process.”
A survey of uncertainty in deep neural networks (2021, unknown): “of aleatoric uncertainty [139], [140], [141].”
Deep Neural Network Concepts for Background Subtraction: A Systematic Review and Comparative Evaluation (2018, background): “ollowing categories: restricted Boltzmann machines (RBMs), deep belief networks(DBNs),autoencoders(AEs)networkanddeepConvolutionalNeuralNetwork (CNNs). In addition, deep probabilistic neural networks [58], deep fuzzy neural networks [46][54] and Generative Adversarial Networks (GANs) [50][162] can a”