Bayesian Hypernetworks

By David Krueger, Chin-Wei Huang, Riashat Islam, Ryan D. Turner, Alexandre Lacoste, Aaron C. Courville. 2017

157 citations recorded by the source corpus.

We propose Bayesian hypernetworks: a framework for approximate Bayesian inference in neural networks. A Bayesian hypernetwork, h, is a neural network which learns to transform a simple noise distribution, p(e) = N(0,I), to a distribution q(t) := q(h(e)) over the parameters t of another neural network (the ``primary network). We train q with variational inference, using an invertible h to enable efficient estimation of the variational lower bound on the posterior p(t | D) via sampling. In contrast to most methods for Bayesian deep learning, Bayesian hypernets can represent a complex multimodal approximate posterior with correlations between parameters, while enabling cheap iid sampling of q(t). In practice, Bayesian hypernets provide a better defense against adversarial examples than dropout, and also exhibit competitive performance on a suite of tasks which evaluate model uncertainty, including regularization, active learning, and anomaly detection.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

Citations in the corpus

Editing Factual Knowledge in Language Models (2021, methodology): “Thus, we apply a trick similar to Krueger et al. (2017) to make g tractably predict edits for modern large deep neural networks ( e.g. , BERT).”
Meta-Learning with Latent Embedding Optimization (2018, background): “Related work has also explored the use of one neural network to produce (some fraction of) the parameters of another (Ha et al., 2016; Krueger et al., 2017), with some approaches focusing on the goal of fast adaptation.”
Weight Agnostic Neural Networks (2019, background): “While the parameters of this distribution can be learned [29, 54], the number of parameters is often greater than the number of weights.”
Uncertainty in Multitask Transfer Learning (2018, methodology): “Recently, signiﬁcant progress has been made to scale Bayesian neural networks to large tasks and to provide better approximations of the posterior distribution [5, 23, 21].”
Posterior Meta-Replay for Continual Learning (2021, background): “However, due to the complexity of modelling high-dimensional distributions at the scale of modern deep learning, BNNs still face severe scalability issues [68].”
Neural Autoregressive Flows (2018, methodology): “While CWN rescales the weight vectors normalized to have unit L2 norm, here we rescale the weight vector normalized by softmax such that it sums to one and is positive.”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, background): “…variational distribution to approximate posterior distribution of (1) the latent representations (Rezende & Mohamed, 2015; Kingma et al., 2016; Tomczak & Welling, 2016; Berg et al., 2018) and (2) the parameters of neural networks (Louizos & Welling, 2017; Krueger et al., 2017; Huang et al., 2019).”
Stochastic Conditional Generative Networks with Basis Decomposition (2019, background): “Followup works like Bayesian Hypernetworks [14] further introduce uncertainty to the generated parameters.”
A survey of uncertainty in deep neural networks (2021, unknown): “The most important approaches for modeling this separation are Bayesian inference [30], [20], [37], [9], [38], ensemble approaches [31], [39], [40], test time data augmentation approaches [41], [42], or single deterministic networks containing explicit components to represent the model and the data ”
Deep Prior (2017, unknown): “1 Learning a Rich Prior Bayesian Neural Networks [1, 2, 3, 4] are now scalable and can be used to estimate prediction uncertainty and model uncertainty [5].”
Adversarial Distillation of Bayesian Neural Network Posteriors (2018, methodology): “Although Hendrycks & Gimpel (2016) and Krueger et al. (2017) showed reasonable results, the deterministic base-line NN already performed very well on this task (i.e., > 90% ).”
HyperGAN: A Generative Model for Diverse, Performant Neural Networks (2019, methodology): “The original hypernetwork produced the target weights as a deterministic function of its own weights, but Bayesian Hypernetworks (BHNs) (Krueger et al., 2017), and Multi-plicative Normalizing Flows (MNF) (Louizos & Welling, 2016) scale and shift model parameters by transforming samples from a…”
Block Neural Autoregressive Flow (2019, methodology): “However, the conditioner grows quadratically with the size of the trans-former network and a combination of restricting the size of the transformer and a technique similar to conditional weight normalization (Krueger et al., 2017) is necessary to reduce the number of parameters.”
Implicit Weight Uncertainty in Neural Networks (2017, methodology): “In contrast to [9], our method works with arbitrary HyperNetworks and relies on density ratio estimation for variational inference with implicit distributions.”
Subspace Inference for Bayesian Deep Learning (2019, background): “Alternatively, both Titsias (2017) and Krueger et al. (2017) propose Bayesian versions of hyper-networks to store meta-models of parameters.”
Bayesian Meta Sampling for Fast Uncertainty Adaptation (2020, methodology): “Note the idea of using NIAF to generate network parameter is similar to the hypernetwork (Ha et al., 2016; Krueger et al., 2017).”