Multiplicative Normalizing Flows for Variational Bayesian Neural Networks

By Christos Louizos, Max Welling. 2017

493 citations recorded by the source corpus.

We reinterpret multiplicative noise in neural networks as auxiliary random variables that augment the approximate posterior in a variational setting for Bayesian neural networks. We show that through this interpretation it is both efficient and straightforward to improve the approximation by employing normalizing flows (Rezende & Mohamed, 2015) while still allowing for local reparametrizations (Kingma et al., 2015) and a tractable lower bound (Ranganath et al., 2015; Maal0e et al., 2016). In experiments we show that with this new approximation we can significantly improve upon classical mean field for Bayesian neural networks on both predictive accuracy as well as predictive uncertainty.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Probabilistic & Bayesian Methods

Citations in the corpus

Stochastic Normalizing Flows (2020, background): “Moreover, having access to tractable density is useful in variational inference (Rezende & Mohamed, 2015; Tomczak & Welling, 2016; Louizos & Welling, 2017; Berg et al., 2018) or approximate sampling from distributions given by an energy function (Oord et al., 2017).”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, background): “The key idea in the de-sign of a ﬂow model is to form f by stacking individual simple invertible transformations (Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018; Rezende & Mohamed, 2015; Kingma et al., 2016; Louizos & Welling, 2017).”
Evidential Deep Learning to Quantify Classification Uncertainty (2018, methodology): “We have also studied the setup suggested in [24], which uses a subset of the classes in CIFAR10 for training and the rest for out-of-distribution uncertainty testing.”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, background): “Recent key advances [31, 47] in variational inference for deep learning typically focus on smaller-scale datasets and architectures.”
Conditional Neural Processes (2018, background): “Finally, the latent variant of CNP can also be seen as an approximated amortized version of Bayesian DL (Gal & Ghahramani, 2016; Blundell et al., 2015; Louizos et al., 2017; Louizos & Welling, 2017) 4.”
Equivariant Flows: exact likelihood generative learning for symmetric densities (2020, background): “…a data sample, having access to tractable density is especially useful in variational inference (Rezende & Mohamed, 2015; Tomczak & Welling, 2016; Louizos & Welling, 2017; Berg et al., 2018) or approximate sampling from distributions given by an energy function (Oord et al., 2017), which can be…”
Uncertainty in Multitask Transfer Learning (2018, background): “[21] and Louizos and Welling [23], but we will soon show that it can be reduced to a simpler architecture in the context of multi-task learning.”
Bayesian Hypernetworks (2017, methodology): “This allows us to overcome the computational limitations of naively-parametrized BHNs noted by Louizos and Welling (2017), since computation now scales linearly, instead of quadratically, in the number of primary net units.”
Posterior Meta-Replay for Continual Learning (2021, background): “Multiple studies have already investigated the use of weight generators for BNNs [33, 22], ranging from sample-based estimates of the prior-matching term [60] to the use of normalizing ﬂows with shared inﬂuence on a set of weights for improved scalability [45].”
Continual learning with hypernetworks (2019, background): “This restriction does not apply to hypernetworks, which can in principle model complex multimodal weight posteriors [19, 12].”
Neural Processes (2018, methodology): “…model in Figure 2d we can write out the conditional marginal likelihood as a hierarchical inference problem: Another interesting area that connects Bayesian methods and NNs are Bayesian neural networks (Gal & Ghahramani, 2016; Blundell et al., 2015; Louizos et al., 2017; Louizos & Welling, 2017).”
Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (2020, background): “In principle, BNNs can permit graceful failure, signalling when a model does not know what to predict (Kendall & Gal, 2017; Dusenberry et al., 2019), and can also generalize better to out-of-distribution examples (Louizos & Welling, 2017; Malinin & Gales, 2018).”
Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks (2020, methodology): “Many methods to achieve this goal have been proposed, both Bayesian (e.g. Blundell et al., 2015; Louizos & Welling, 2017; Zhang et al., 2018) and non-Bayesian (e.g. Lakshminarayanan et al., 2017; Liang et al., 2018; Hein et al., 2019).”
Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples (2017, methodology): “As another line of research, Bayesian probabilistic models [10, 12] and ensembles [9] were also investigated.”
Lifelong Generative Modeling (2017, background): “requirement for the parameters of a neural network and there is active research in Bayesian neural networks that attempts to relax this constraint [74, 75, 80].”
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019, methodology): “…trained independently on the entire dataset using random initialization (Lakshminarayanan et al., 2017) (we set M = 10 in experiments below) • ( SVI ) Stochastic Variational Bayesian Inference for deep learning (Blundell et al., 2015; Graves, 2011; Louizos & Welling, 2017, 2016; Wen et al., 2018).”
Bayesian Compression for Deep Learning (2017, background): “We also found that constraining the standard deviations as described at [46] and “warm-up" [65] helps in avoiding bad local optima of the variational objective.”
Do Deep Generative Models Know What They Don't Know? (2018, background): “For example, Louizos & Welling (2017) show that simply rotating an MNIST digit can make a neural network predict another class with high conﬁdence (see their Figure 1a).”
Neural Spline Flows (2019, background): “In recent years, normalizing flows have received widespread attention in the machine-learning literature, seeing successful use in density estimation [9, 41], variational inference [29, 35, 44, 55], image, audio and video generation [25, 27, 31, 43], likelihood-free inference [42], and learning maxi”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “, 2016; Tomczak and Welling, 2016) and global (Louizos and Welling, 2017) variables.”