Implicit Weight Uncertainty in Neural Networks

By Nick Pawlowski, Martin Rajchl, Ben Glocker. 2017

100 citations recorded by the source corpus.

We interpret HyperNetworks within the framework of variational inference within implicit distributions. Our method, Bayes by Hypernet, is able to model a richer variational distribution than previous methods. Experiments show that it achieves comparable predictive performance on the MNIST classification task while providing higher predictive uncertainties compared to MC-Dropout and regular maximum likelihood training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Posterior Meta-Replay for Continual Learning (2021, background): “Note, also Pawlowski et al. (2017) reports difficulties when applying AVB to BNNs.”
Continual learning with hypernetworks (2019, background): “However, hypernetworks can be invoked iteratively, ﬁlling in only part of the target model at each step, in chunks (Ha et al., 2017; Pawlowski et al., 2017).”
Principled Weight Initialization for Hypernetworks (2020, methodology): “, 2018), Bayesian neural networks (Krueger et al., 2017; Ukai et al., 2018; Pawlowski et al., 2017; Henning et al., 2018; Deutsch et al., 2019), multi-task learning (Pan et al.”
Bayesian Meta Sampling for Fast Uncertainty Adaptation (2020, methodology): “Using hypernetworks to model posterior distributions have also been studied in (Pawlowski et al., 2017; Sheikh et al., 2017).”
In Defense of the Learning Without Forgetting for Task Incremental Learning (2021, methodology): “Hyper-CL employs a variant of Hypernetworks [8], called Chunked-Hypernetworks [25], which generates different subsets of the target network’s parameters using the same generator.”
Hypermodels for Exploration (2020, background): “Pawlowski et al. (2017) introduced another variational inference based algorithm that interprets hypernetworks as implicit distributions, i.e. distributions that may have intractable probability density functions but allow for easy sampling.”
Bayesian Layers: A Module for Neural Network Uncertainty (2018, background): “…restriction of independence across layers or tractable densities; hierarchical variational models (Ranganath et al., 2016) and implicit posteriors (Pawlowski et al., 2017) are compatible. we overload the existing parameter regularization arguments in a layer’s constructor; in Keras, it is…”
Learning for Single-Shot Confidence Calibration in Deep Neural Networks Through Stochastic Inferences (2018, background): “However, the exact Bayesian inference is not tractable in deep neural networks due to its high computational cost, and various approximate inference techniques—MCMC [17], Laplace approximation [14] and variational inference [1, 4, 8, 20]—have been proposed.”
Deep learning with noisy labels: exploring techniques and remedies in medical image analysis (2019, background): “Many studies in recent years have aimed at improving the calibration of deep learning models [116], [117], [118].”
Uncertainty-Aware Deep Classifiers Using Generative Models (2020, methodology): “Other approaches are also trained using the same classiﬁer architecture with the priors and posteriors described in (Louizos and Welling 2017) and (Pawlowski et al. 2017).”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, methodology): “[232] proposed a novel technique of variational approximation, termed as Bayes by Hypernet (BbH) that deduced hypernetworks as implicit distributions.”
HyperGAN: A Generative Model for Diverse, Performant Neural Networks (2019, methodology): “(Pawlowski et al., 2017) learn an approximate posterior by using a hypernetwork to generate weight samples from a Gaussian prior.”
Approximating the Predictive Distribution via Adversarially-Trained Hypernetworks (2018, methodology): “They were repurposed and reintroduced multiple times in different forms to approximate the weight posterior while employing different strategies to estimate the so-called prior-matching term [14, 20, 24, 32, 33].”
Model Selection in Bayesian Neural Networks via Horseshoe Priors (2017, background): “They have been explored both in the classic works of (MacKay, 1992; Neal, 1993, 1997) as well as in more recent approaches (Louizos and Welling, 2016; Hernández-Lobato and Adams, 2015; Louizos and Welling, 2017; Pawlowski et al., 2017).”