A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference

By Kumar Shridhar, Felix Laumann, Marcus Liwicki. 2019

225 citations recorded by the source corpus.

Artificial Neural Networks are connectionist systems that perform a given task by learning on examples without having prior knowledge about the task. This is done by finding an optimal point estimate for the weights in every node. Generally, the network using point estimates as weights perform well with large datasets, but they fail to express uncertainty in regions with little or no data, leading to overconfident decisions. In this paper, Bayesian Convolutional Neural Network (BayesCNN) using Variational Inference is proposed, that introduces probability distribution over the weights. Furthermore, the proposed BayesCNN architecture is applied to tasks like Image Classification, Image Super-Resolution and Generative Adversarial Networks. The results are compared to point-estimates based architectures on MNIST, CIFAR-10 and CIFAR-100 datasets for Image CLassification task, on BSD300 dataset for Image Super Resolution task and on CIFAR10 dataset again for Generative Adversarial Network task. BayesCNN is based on Bayes by Backprop which derives a variational approximation to the true posterior. We, therefore, introduce the idea of applying two convolutional operations, one for the mean and one for the variance. Our proposed method not only achieves performances equivalent to frequentist inference in identical architectures but also incorporate a measurement for uncertainties and regularisation. It further eliminates the use of dropout in the model. Moreover, we predict how certain the model prediction is based on the epistemic and aleatoric uncertainties and empirically show how the uncertainty can decrease, allowing the decisions made by the network to become more deterministic as the training accuracy increases. Finally, we propose ways to prune the Bayesian architecture and to make it more computational and time effective.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Probabilistic & Bayesian Methods

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “Bayesian neural networks [358] have been proposed and shown to be far more robust to over-fitting, and can easily learn from small datasets.”
Continual Deep Learning by Functional Regularisation of Memorable Past (2020, background): “Methods such as VCL+coresets perform rather poorly, which is likely because Bayes By Backprop (Blundell et al., 2015) is difﬁcult to train on CNNs (Osawa et al., 2019; Shridhar et al., 2019).”
Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference (2018, methodology): “Recently, the uncertainty afforded by Bayes by Backprop trained neural networks has been used successfully to train feedforward neural networks in both supervised and reinforcement learning environments [5, 7, 8], for training recurrent neural networks [9], and for CNNs [10, 11].”
Bayesian Convolutional Neural Networks (2018, methodology): “networks has been used successfully to train feedforward neural networks in both supervised and reinforcement learning environments [5, 7, 8], for training recurrent neural networks [9], and for CNNs [10, 11]. Here, we review this method for CNNs to construct a common foundation on which be build on”