Variational Gaussian Dropout is not Bayesian

By Jiri Hron, Alexander G. de G. Matthews, Zoubin Ghahramani. 2017

49 citations recorded by the source corpus.

Gaussian multiplicative noise is commonly used as a stochastic regularisation technique in training of deterministic neural networks. A recent paper reinterpreted the technique as a specific algorithm for approximate inference in Bayesian neural networks; several extensions ensued. We show that the log-uniform prior used in all the above publications does not generally induce a proper posterior, and thus Bayesian inference in such models is ill-posed. Independent of the log-uniform prior, the correlated weight noise approximation has further issues leading to either infinite objective or high risk of overfitting. The above implies that the reported sparsity of obtained solutions cannot be explained by Bayesian or the related minimum description length arguments. We thus study the objective from a non-Bayesian perspective, provide its previously unknown analytical form which allows exact gradient evaluation, and show that the later proposed additive reparametrisation introduces minima not present in the original multiplicative parametrisation. Implications and future research directions are discussed.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling (2018, methodology): “Dropout can be seen as optimizing a variational objective (Kingma et al., 2015; Gal & Ghahramani, 2016; Hron et al., 2017).”
Variance Networks: When Expectation Does Not Meet Your Expectations (2018, background): “…| ν)) = (9) = const− 1 2 logαµ2 + ν + 1 2 Eε∼N (0,1) log ( ν + µ2(1 + √ αε)2 ) (10) As the use of the improper log-uniform prior in neural networks is questionable (Hron et al. (2017)), we argue that the Student’s t-distribution with diminishing values of ν results in the same properties of…”
Variational Bayesian dropout: pitfalls and fixes (2018, background): “An earlier version of this work was published in (Hron et al., 2017).”
Bayesian Gradient Descent: Online Variational Bayes Learning with Increased Robustness to Catastrophic Forgetting and Weight Pruning (2018, background): “Moreover, later, (Hron et al., 2017) noted that such variational dropout based approaches are not Bayesian.”
Hypermodels for Exploration (2020, background): “Our motivation is that intelligent ∗DeepMind (1)Although later work suggests that this dropout approximation can be of poor quality (Osband, 2016; Hron et al., 2017).”