Fixing Variational Bayes: Deterministic Variational Inference for Bayesian Neural Networks

By Anqi Wu, Sebastian Nowozin, Edward Meeds, Richard E. Turner, José Miguel Hernández-Lobato, Alexander L. Gaunt. 2018

16 citations recorded by the source corpus.

Bayesian neural networks (BNNs) hold great promise as a flexible and principled solution to deal with uncertainty when learning from finite data. Among approaches to realize probabilistic inference in deep neural networks, variational Bayes (VB) is theoretically grounded, generally applicable, and computationally efficient. With wide recognition of potential advantages, why is it that variational Bayes has seen very limited practical use for BNNs in real applications? We argue that variational inference in neural networks is fragile: successful implementations require careful initialization and tuning of prior variances, as well as controlling the variance of Monte Carlo gradient estimates. We fix VB and turn it into a robust inference tool for Bayesian neural networks. We achieve this with two innovations: first, we introduce a novel deterministic method to approximate moments in neural networks, eliminating gradient variance; second, we introduce a hierarchical prior for parameters and a novel empirical Bayes procedure for automatically selecting prior variances. Combining these two innovations, the resulting method is highly efficient and robust. On the application of heteroscedastic regression we demonstrate strong predictive performance over alternative approaches.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, methodology): “Additionally, we note the strong performance of well-tuned SGD as a baseline against the other approximate inference methods, as it consistently performs nearly as well as both SWAG and DVI.”
Subspace Inference for Bayesian Deep Learning (2019, methodology): “The subspace methods outperform SGD and DVI on naval, concrete and yacht and are competitive on the other datasets.”
Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights (2020, background): “…area, and the main focuses of which are (i) improving the reparameterization gradient estimator to enable faster convergence [Miller et al., 2017, Wu et al., 2018], (ii) replacing the typical standard Normal prior, p(w) = N (w;0, I) by a structured prior that better models the structures present…”
Bayesian Layers: A Module for Neural Network Uncertainty (2018, methodology): “, 2018), and deterministic variational inference (Wu et al., 2018).”