Overpruning in Variational Bayesian Neural Networks

By Brian L. Trippe, Richard E. Turner. 2018

55 citations recorded by the source corpus.

The motivations for using variational inference (VI) in neural networks differ significantly from those in latent variable models. This has a counter-intuitive consequence; more expressive variational approximations can provide significantly worse predictions as compared to those with less expressive families. In this work we make two contributions. First, we identify a cause of this performance gap, variational over-pruning. Second, we introduce a theoretically grounded explanation for this phenomenon. Our perspective sheds light on several related published results and provides intuition into the design of effective variational approximations of neural networks.

Subfields: Artificial Intelligence, Efficiency & Compression, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

In Probabilistic & Bayesian Methods

Citations in the corpus

Meta-Learning Probabilistic Inference for Prediction (2018, background): “This is due to VI’s tendency to under-fit, especially for small numbers of data points (Trippe and Turner, 2018; Turner and Sahani, 2011) which is compounded when using inference networks (Cremer et al.”
Generalized Variational Continual Learning (2020, methodology): “When applied to multi-head networks, VCL tends to prune out large parts of the network (Trippe & Turner, 2018; Turner & Sahani, 2011) and GVCL inherits this behaviour.”
Improving and Understanding Variational Continual Learning (2019, methodology): “Previous work on Variational Inference for BNNs has found that VI methods can be used to prune large parts of the network [12], and we show here how this pruning is done over units as opposed to weights (see also Trippe and Turner [20]).”
Task Agnostic Continual Learning Using Online Variational Bayes (2018, background): “BGD does not experience underfitting and over-pruning, as shown by [32] for BBB.”
Task-Agnostic Continual Learning Using Online Variational Bayes With Fixed-Point Updates (2020, background): “Thus, FOO-VB does not experience underfitting and overpruning as Trippe and Turner (2018) showed for BBB (Blundell et al., 2015).”
Bayesian Gradient Descent: Online Variational Bayes Learning with Increased Robustness to Catastrophic Forgetting and Weight Pruning (2018, background): “Indeed, Trippe and Turner (2018) showed that variational Bayesian neural networks, trained using BBB, suffer from variational over-pruning.”
Kernel Implicit Variational Inference (2017, methodology): “This problem has been pointed out previously, both in VAE works (Sønderby et al., 2016; Hoffman, 2017) and BNN works (Blundell et al., 2015), and can be explained by the looseness of the variational bound when factorized approximation is used (Trippe & Turner, 2017).”
Doubly Semi-Implicit Variational Inference (2018, background): “In Bayesian neural networks, a standard normal prior is the default choice, but together with the mean field posterior, it can lead to overpruning and consequently underfitting [39].”
Continual Learning with Bayesian Neural Networks for Non-Stationary Data (2020, background): “Previous work hypothesised that this problem is an artifact of the ELBO and not an optimisation problem (Trippe & Turner, 2018; Turner & M. Sahani, 2011).”
Indian Buffet Neural Networks for Continual Learning (2019, background): “Note that the sparsity induced by the IBP prior renders the effects of variational overpruning redundant [21].”
Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights (2020, unknown): “Note that the mean-field variational Gaussian approximation with a standard normal prior, presented in is often outperformed by point estimation in certain settings [Trippe and Turner, 2018].”