The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
By Jakub Bartlomiej Swiatkowski, Kevin Roth, Bastiaan S. Veeling, Linh Hoang Tran, Joshua V. Dillon, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, Sebastian Nowozin. 2020
55 citations recorded by the source corpus.
Variational Bayesian Inference is a popular methodology for approximating posterior distributions over Bayesian neural network weights. Recent work developing this class of methods has explored ever richer parameterizations of the approximate posterior in the hope of improving performance. In contrast, here we share a curious experimental finding that suggests instead restricting the variational distribution to a more compact parameterization. For a variety of deep Bayesian neural networks trained using Gaussian mean-field variational inference, we find that the posterior standard deviations consistently exhibit strong low-rank structure after convergence. This means that by decomposing these variational parameters into a low-rank factorization, we can make our variational approximation more compact without decreasing the models' performance. Furthermore, we find that such factorized parameterizations improve the signal-to-noise ratio of stochastic gradient estimates of the variational lower bound, resulting in faster convergence.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dustin Tran
- Jasper Snoek
- Balaji Lakshminarayanan
- Michael W. Dusenberry
- Yeming Wen
- Katherine A. Heller
- Ghassen Jerfel
- Sergey Levine
In Computer Vision and Pattern Recognition
- Jasper Snoek
- Dustin Tran
- Balaji Lakshminarayanan
- Michael W. Dusenberry
- Yeming Wen
- Paul W. Fieguth
- Abbas Khosravi
- Katherine A. Heller
Citations in the corpus
- Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (2020, background): “Swiatkowski et al. (2019) apply singular value decomposition post-training and observe that a rank of 1-3 captures most of the variational posterior’s variance.”
- A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, methodology): “7: A summary of various VI methods for BDL which is reproduced based on [84].”