Variance Networks: When Expectation Does Not Meet Your Expectations

By Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, Dmitry P. Vetrov. 2018

25 citations recorded by the source corpus.

Ordinary stochastic neural networks mostly rely on the expected values of their weights to make predictions, whereas the induced noise is mostly used to capture the uncertainty, prevent overfitting and slightly boost the performance through test-time averaging. In this paper, we introduce variance layers, a different kind of stochastic layers. Each weight of a variance layer follows a zero-mean distribution and is only parameterized by its variance. We show that such layers can learn surprisingly well, can serve as an efficient exploration tool in reinforcement learning tasks and provide a decent defense against adversarial attacks. We also show that a number of conventional Bayesian neural networks naturally converge to such zero-mean posteriors. We observe that in these cases such zero-mean parameterization leads to a much better training objective than conventional parameterizations where the mean is being learned.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Weight Agnostic Neural Networks (2019, background): “Recently, Neklyudov et al. [79] proposed variance networks, which sample each weight from a distribution with a zero mean and a learned variance parameter, and show that ensemble evaluations can improve performance on image recognition tasks.”
A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “Following [32, 51], we do not sample the weights w , but we sample instead layer activations b due to its consequent computational acceleration.”
Averaging Weights Leads to Wider Optima and Better Generalization (2018, background): “In recent work Neklyudov et al. [2018] also discuss an efﬁcient approach for model averaging of Bayesian neural networks.”
Bayesian Convolutional Neural Networks (2018, methodology): “Following [10, 18], we do not sample the weights w, but we sample instead layer activations b due to its”
Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference (2018, methodology): “Recently, the uncertainty afforded by Bayes by Backprop trained neural networks has been used successfully to train feedforward neural networks in both supervised and reinforcement learning environments [5, 7, 8], for training recurrent neural networks [9], and for CNNs [10, 11].”
A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks (2019, background): “Despite BDL’s impact on a range of real-world applications and the flourish of recent ideas and inference techniques [11, 3, 8, 48, 50, 37], the development of the field itself is impeded by the lack of realistic benchmarks to guide research.”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, background): “[166] proposed a different kind of stochastic layer called variance layers.”
Bayesian Convolutional Neural Networks with Variational Inference (2018, methodology): “Following [16, 27], we do not sample the weights w , but we sample instead layer activations b due to its consequent computational acceleration.”