Assumed Density Filtering Methods for Learning Bayesian Neural Networks

By Soumya Ghosh, F. D. Fave, Jonathan S. Yedidia. 2016

56 citations recorded by the source corpus.

Buoyed by the success of deep multilayer neural networks, there is renewed interest in scalable learning of Bayesian neural networks. Here, we study algorithms that utilize recent advances in Bayesian inference to efficiently learn distributions over network weights. In particular, we focus on recently proposed assumed density filtering based methods for learning Bayesian neural networks -- Expectation and Probabilistic backpropagation. Apart from scaling to large datasets, these techniques seamlessly deal with non-differentiable activation functions and provide parameter (learning rate, momentum) free learning. In this paper, we first rigorously compare the two algorithms and in the process develop several extensions, including a version of EBP for continuous regression problems and a PBP variant for binary classification. Next, we extend both algorithms to deal with multiclass classification and count regression problems. On a variety of diverse real world benchmarks, we find our extensions to be effective, achieving results competitive with the state-of-the-art.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

Citations in the corpus

A General Framework for Uncertainty Estimation in Deep Learning (2019, methodology): “ibutions, following the work in [16] for non-linear Gaussian belief networks. Moreover, the distribution is propagated through the network in a single pass using Assumed Density Filtering (ADF) [13], [14] (Fig. 2). The main advantage of this approach is that it is possible to assume a distribution o”
Lightweight Probabilistic Deep Networks (2018, background): “While [14, 21] also apply ADF in a Bayesian context, they in contrast approximate the distribution of weights rather than the activation distributions.”
Fixing Variational Bayes: Deterministic Variational Inference for Bayesian Neural Networks (2018, methodology): “Since the original work, PBP has been generalized to classification (Ghosh et al., 2016) and richer posterior families such as the matrix variate Normal posteriors (Sun et al.”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, methodology): “Accordingly, learning of different BNNs [518], [156] can be optimized using various Assumed Density Filtering (ADF) techniques [519] and apply them for different machine and deep learning tasks (e.”
A Scalable Laplace Approximation for Neural Networks (2018, methodology): “approaches of (Hernández-Lobato & Adams, 2015) and (Ghosh et al., 2016) assume independence between the individual weights which, particularly when optimising the KL divergence, often lets the model underestimate the uncertainty about the weights.”
Deterministic Variational Inference for Robust Bayesian Neural Networks (2018, background): “Since the original work, PBP has been generalized to classification (Ghosh et al., 2016) and richer posterior families such as the matrix variate Normal posteriors (Sun et al.”