Functional Variational Bayesian Neural Networks

By Guodong Zhang, Jiaxin Shi, Roger Baker Grosse. 2019

274 citations recorded by the source corpus.

Variational Bayesian neural networks (BNNs) perform variational inference over weights, but it is difficult to specify meaningful priors and approximate posteriors in a high-dimensional weight space. We introduce functional variational Bayesian neural networks (fBNNs), which maximize an Evidence Lower BOund (ELBO) defined directly on stochastic processes, i.e. distributions over functions. We prove that the KL divergence between stochastic processes equals the supremum of marginal KL divergences over all finite sets of inputs. Based on this, we introduce a practical training objective which approximates the functional ELBO using finite measurement sets and the spectral Stein gradient estimator. With fBNNs, we can specify priors entailing rich structures, including Gaussian processes and implicit stochastic processes. Empirically, we find fBNNs extrapolate well using various structured priors, provide reliable uncertainty estimates, and scale to large datasets.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Continual Learning via Sequential Function-Space Variational Inference (2023, background): “To address the fact that the KL divergence between distributions over functions is not in general tractable, prior works have developed estimation procedures that allow turning Equation (3) into an objective function that can be used in practice (Rudner et al., 2021; Sun et al., 2019).”
Sliced Score Matching: A Scalable Approach to Density and Score Estimation (2019, background): “Implicit distributions can arise in many more situations such as the marginal distribution of a non-conjugate model (Sun et al., 2019), and models deﬁned by complex simulation processes (Tran et al., 2017).”
Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, background): “, 2019), function-space inspired priors (e.g., Yang et al., 2019; Louizos et al., 2019; Sun et al., 2019; Hafner et al., 2018), and developing flat objective priors in parameter space, directly leveraging the biases of the neural network functional form (e.”
Rethinking Function-Space Variational Inference in Bayesian Neural Networks (2020, methodology): “We evaluate the proposed function-space variational inference ( fsvi ) method on illustrative regression and classiﬁcation tasks as well as on high-dimensional classiﬁcation tasks prior work (Sun et al., 2019) was unable to scale to.”
Bayesian Few-Shot Classification with One-vs-Each Pólya-Gamma Augmented Gaussian Processes (2020, background): “Moreover, specifying meaningful priors in parameter space is known to be difficult due to the complex relationship between weights and functions in deep networks [29].”
A survey of uncertainty in deep neural networks (2021, background): “filtering [129] etc, (ii) probabilistic programming to exploit modern Graphical Processing Units (GPUs) [130], [131], [132], [133], (iii) different types of priors [134], [135], (iv) advancements in theoretical understandings of BNNs [136], [121], [137], (iv) uncertainty propagation techniques to sp”
Model Selection in Bayesian Neural Networks via Horseshoe Priors (2017, background): “More broadly, model selection via Horseshoe priors may represent a sweet spot, at least for now, between (still the most common) fully factorized priors and recent attempts at specifying function-based priors (Wang et al., 2019; Sun et al., 2019).”
Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences (2020, methodology): “Bayesian neural networks typically either perform Markov Chain Monte Carlo (MCMC) sampling (MacKay, 1992), variational inference (Sun et al., 2019; Khan et al., 2018), or use hybrid methods such as particle-based inference (Liu & Wang, 2016) to approximate the posterior distribution over neural…”
Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning (2019, background): “Louizos and Welling [2016], Sun et al. [2017] and Oh et al. [2019] have all introduced richer variational distributions which permit correlations between weights to be learned by the BNN; Sun et al. [2019] instead perform inference in function-space.”
Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing (2019, background): “20 IEEE SIGNAL PROCESSING MAGAZINE | March 2021 | deep networks [7]–[11].”
Drug discovery with explainable artificial intelligence (2020, background): “152,153 Gal et al. , 154 suggested the usage of dropout regularization to perform approximate Bayesian inference, which was later extended 155 to compute epistemic ( i.e. , caused by model mis-speciﬁcation) and aleatory uncertainty (inherent to the noise in the data).”
Action and Perception as Divergence Minimization (2020, methodology): “…includes hierarchical priors (Louizos and Welling, 2016; Ghosh and Doshi-Velez, 2017), data priors (Louizos and Welling, 2016; Hafner et al., 2019b; Sun et al., 2019), ﬂexible posteriors (Louizos and Welling, 2016; Sun et al., 2017; Louizos and Welling, 2017; Zhang et al., 2018; Chang et al.,…”
Variational Implicit Processes (2018, methodology): “n inverse of VIP by encoding properties of GP priors into BNN weight priors, which is then used to regularize BNN inference. This idea is further investigated by a concurrent work on functional BNNs (Sun et al., 2018), where GP priors are directly used to regularize BNN training through gradient est”
Noise Contrastive Priors for Functional Uncertainty (2018, background): “This is similar in nature to Sun et al. (2019), who define a GP prior for BNNs resulting in an interesting but more complex algorithm.”
Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights (2020, methodology): “s and activation functions to induce desired properties through architectural choices, but is not expressing these assumptions as a weight prior. The functional regularization approach introduced in [Sun et al., 2019] shares some of the motivations with our paper, but implements it very dierently by”
Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited (2020, background): “Alternatively, by working directly in function space, as in Sun et al. (2019), the redundancy of many parameters could be avoided.”
A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks (2019, methodology): “[4] produced their analysis, while less work has gone into building on the methodology of the more computationally parsimonious Bayesian deep learning approaches like deep ensembles or MC dropout [32, 44, 45, 38].”