Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors

By Christos Louizos, Max Welling. 2016

276 citations recorded by the source corpus.

We introduce a variational Bayesian neural network where the parameters are governed via a probability distribution on random matrices. Specifically, we employ a matrix variate Gaussian (Gupta & Nagar, 1999) parameter posterior distribution where we explicitly model the covariance among the input and output dimensions of each layer. Furthermore, with approximate covariance matrices we can achieve a more efficient way to represent those correlations that is also cheaper than fully factorized parameter posteriors. We further show that with the "local reprarametrization trick" (Kingma et al., 2015) on this posterior distribution we arrive at a Gaussian Process (Rasmussen, 2006) interpretation of the hidden units in each layer and we, similarly with (Gal & Ghahramani, 2015), provide connections with deep Gaussian processes. We continue in taking advantage of this duality and incorporate "pseudo-data" (Snelson & Ghahramani, 2005) in our model, which in turn allows for more efficient posterior sampling while maintaining the properties of the original model. The validity of the proposed approach is verified through extensive experiments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Probabilistic & Bayesian Methods

Citations in the corpus

Overcoming Catastrophic Forgetting by Incremental Moment Matching (2017, methodology): “We think it could be alleviated in future work by using a more complex prior, such as a matrix Gaussian distribution considering the correlations between nodes in the network [31].”
Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (2020, background): “The outer product of rank-1 vectors resembles matrixvariate Gaussians (Louizos & Welling, 2016): the major difference is that rank-1 priors are uncertain about the scale factors shared across rows and columns rather than ﬁxing a covariance.”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, background): “, 2015; Hernández-Lobato and Adams, 2015) or improving its posterior approximation (Louizos and Welling, 2016; Zhang et al., 2017; Krueger et al., 2017).”
Predictive Uncertainty Estimation via Prior Networks (2018, methodology): “) using Bayes’ rule is intractable, and it is necessary to use either an explicit or implicit variational approximation q ( θ ) [25, 26, 27, 28]: Furthermore, the integral in eq.”
Lifelong Generative Modeling (2017, background): “requirement for the parameters of a neural network and there is active research in Bayesian neural networks that attempts to relax this constraint [74, 75, 80].”
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019, methodology): “…trained independently on the entire dataset using random initialization (Lakshminarayanan et al., 2017) (we set M = 10 in experiments below) • ( SVI ) Stochastic Variational Bayesian Inference for deep learning (Blundell et al., 2015; Graves, 2011; Louizos & Welling, 2017, 2016; Wen et al., 2018).”
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (2018, methodology): “The goal of this approach is similar to other approaches that employ structured approximations (Ritter et al., 2018; Louizos & Welling, 2016; Sun et al., 2017).”
Noisy Natural Gradient as Variational Inference (2017, methodology): “Note that VMG introduced pseudo input-output pairs to enhance the ﬂexibility of posterior distribution.”
Action and Perception as Divergence Minimization (2020, methodology): “This includes hierarchical priors (Louizos and Welling, 2016; Ghosh and Doshi-Velez, 2017), data priors (Louizos and Welling, 2016; Hafner et al., 2019b; Sun et al., 2019), ﬂexible posteriors (Louizos and Welling, 2016; Sun et al., 2017; Louizos and Welling, 2017; Zhang et al., 2018; Chang et al.,…”
Eigenvalue Corrected Noisy Natural Gradient (2018, unknown): “In comparison to existing methods that ﬁt MVG posteriors [Sun et al., 2017, Louizos and Welling, 2016], noisy K-FAC does not assume additional approximations.”
SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient (2018, methodology): “Other types of approximation methods include Bayesian dropout [10] and methods that use matrix-variate Gaussians [21, 33].”
A Scalable Laplace Approximation for Neural Networks (2018, methodology): “The posterior that we obtain is reminiscent of (Louizos & Welling, 2016) and (Sun et al., 2017), who optimise the parameters of a matrix normal distribution as their weights, which requires a modification of the training procedure.”
In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning (2021, methodology): “Uncertainty and Calibration Estimating network prediction uncertainty has been a deeply studied topic (Graves, 2011; Blundell et al., 2015; Louizos & Welling, 2016; Lakshminarayanan et al., 2017; Malinin & Gales, 2018; Maddox et al., 2019; Welling & Teh, 2011).”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, unknown): “Louizos and Welling [158] used MVG distribution to estimate the weight posterior uncertainty.”
Learning Neural Networks with Adaptive Regularization (2019, unknown): “As an exception, Louizos and Welling [29], Sun et al.”
Kernel Implicit Variational Inference (2017, methodology): “We compare KIVI with VI using a single Gaussian posterior (Fig.”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “Estimating the uncertainty of neural networks’ prediction has attracted a lot of interest [62, 63], with the majority of works involving Bayesian approaches [64, 62, 65].”