Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference

By Yarin Gal, Zoubin Ghahramani. 2015

829 citations recorded by the source corpus.

Convolutional neural networks (CNNs) work well on large datasets. But labelled data is hard to collect, and in some applications larger amounts of data are not available. The problem then is how to use CNNs with small data -- as CNNs overfit quickly. We present an efficient Bayesian CNN, offering better robustness to over-fitting on small data than traditional approaches. This is by placing a probability distribution over the CNN's kernels. We approximate our model's intractable posterior with Bernoulli variational distributions, requiring no additional model parameters. On the theoretical side, we cast dropout network training as approximate inference in Bayesian neural networks. This allows us to implement our model using existing tools in deep learning with no increase in time complexity, while highlighting a negative result in the field. We show a considerable improvement in classification accuracy compared to standard techniques and improve on published state-of-the-art results for CIFAR-10.

Subfields: Artificial Intelligence, Model Architectures, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Probabilistic & Bayesian Methods

Citations in the corpus

Evidential Deep Learning to Quantify Classification Uncertainty (2018, methodology): “1 In this section, we compared the following approaches: (a) L2 corresponds to the standard deterministic neural nets with softmax output and weight decay, (b) Dropout refers to the uncertainty estimation model used in [8], (c) Deep Ensemble refers to the model used in [21], (d) FFG refers to”
A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “Speciﬁcally, in Gal’s and Ghahramani’s [13] approach, no prior probability distributions p ( w ) are placed on the Cnn ’s parameters.”
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? (2017, methodology): “Dropout variational inference is a practical approach for approximate inference in large and complex models [15].”
Dropout Inference in Bayesian Neural Networks with Alpha-divergences (2017, methodology): “Given a (deterministic) neural network, stochastic regularisation techniques in the model (such as dropout (Hinton et al., 2012; Srivastava et al., 2014)) can be interpreted as variational Bayesian approximations in a Bayesian NN with the same network structure (Gal & Ghahramani, 2016b).”
Towards Robust Evaluations of Continual Learning (2018, methodology): “For a simple demonstration, we train a network with two convolutional layers with dropout (3x3 kernels and 32 then 64 channels) (Gal and Ghahramani, 2016), a”
Information Dropout: Learning Optimal Representations Through Noisy Computation (2016, neutral): “(3) (see also [15]).”
Deep Bayesian Active Learning with Image Data (2017, methodology): “Taking advantage of specialised models such as Bayesian convolutional neural networks (BCNNs) (Gal & Ghahramani, 2016a;b), we demonstrate our active learning techniques with image data.”
Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding (2015, methodology): “In [12] it was shown that minimising the cross entropy loss objective function has the effect of minimising the Kullback-Leibler divergence term.”
Learning 3D Object Categories by Looking Around Them (2017, methodology): “Following [18], for training it uses the BerHu depth loss and a dropout layer, which allows it to produce a confidence score of the depth measurements at test time using the sampling technique of [17, 9].”
Leveraging uncertainty information from deep neural networks for disease detection (2016, methodology): “Due to its ease of use and inherent scalability, a recent insight from Gal & Ghahramani 30,31,34 is particularly promising for use in medical settings.”
Swapout: Learning an ensemble of deep architectures (2016, background): “Note that such stochastic inference with dropout has been studied in [2].”
Analyzing Inverse Problems with Invertible Neural Networks (2018, background): “Neural networks can be trained to predict accurate sufficient statistics for parametric posteriors [30, 39], or can be designed to learn a mean-field distribution for the network’s weights via dropout variational inference [10, 24].”
Decoupled Certainty-Driven Consistency Loss for Semi-supervised Learning (2019, background): “One of them is Monte Carlo dropout proposed by [9, 10].”
A Survey of Deep Active Learning (2020, methodology): “To address the first problem, some researchers have applied Bayesian DL [70] to deal with the high-dimensional mini-batch samples with fewer queries in the AL context [72, 115, 165, 223], thereby effectively alleviating the problem of the DL model being too confident about the output results.”