Probabilistic Meta-Representations Of Neural Networks

By Theofanis Karaletsos, Peter S. Dayan, Zoubin Ghahramani. 2018

29 citations recorded by the source corpus.

Existing Bayesian treatments of neural networks are typically characterized by weak prior and approximate posterior distributions according to which all the weights are drawn independently. Here, we consider a richer prior distribution in which units in the network are represented by latent variables, and the weights between units are drawn conditionally on the values of the collection of those variables. This allows rich correlations between related weights, and can be seen as realizing a function prior with a Bayesian complexity regularizer ensuring simple solutions. We illustrate the resulting meta-representations and representations, elucidating the power of this prior.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Andrew Gordon Wilson
2. Pavel Izmailov
3. Ian Osband
4. Benjamin Van Roy
5. Dmitry P. Vetrov
6. Wesley J. Maddox
7. Zheng Wen
8. Polina Kirichenko

In Statistical and Nonlinear Physics

1. Thang D. Bui
2. Pavel Izmailov
3. Johannes von Oswald
4. Andrew Gordon Wilson
5. Wesley J. Maddox
6. Benjamin Van Roy
7. Christian Henning
8. Dmitry P. Vetrov

Citations in the corpus

• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights (2020, methodology): “…inference scheme that makes use of innovations from inducing point GP approximation literature [Titsias, 2009, Hensman et al., 2013, Quiñonero-Candela and Rasmussen, 2005, Matthews et al., 2016, Bui et al., 2017] and previous work on inferring meta-representations [Karaletsos et al., 2018].”
• Hypermodels for Exploration (2020, background): “Karaletsos et al. (2018) studied Bayesian neural networks with correlated priors, specifically considering prior distributions in which units in the neural network are represented by latent variables and weights between units are drawn conditionally on the values of those latent variables.”
• Subspace Inference for Bayesian Deep Learning (2019, background): “Similarly, Karaletsos et al. (2018) propose to use a metaprior in a low-dimensional space to perform variational inference for BNNs.”
• Approximating the Predictive Distribution via Adversarially-Trained Hypernetworks (2018, methodology): “An interesting method has been recently proposed by Karaletsos et al. [14] by shifting the problem of performing VI to the input space of the hypernetwork .”