Weight Uncertainty in Neural Network

By Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, Daan Wierstra. 2015

2,425 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

On The Impact of Machine Learning Randomness on Group Fairness (2023, background): “[11] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.”
Overcoming catastrophic forgetting in neural networks (2016, methodology): “e (as in a Laplace approximation) does constitute a signiﬁcant weakness (see Fig 4C). Our initial explorations suggest that one might improve on this local estimate by using Bayesian neural networks [Blundell et al., 2015]. While this paper has primarily focused on building an algorithm out of neuro”
Repurposing Pretrained Models for Robust Out-of-domain Few-Shot Learning (2021, background): “For each parameter, UCB scales its stepsize inversely proportional to the uncertainty of the parameter in the BNN to reduce changes in important parameters while allowing less important parameters to be modiﬁed faster in favor of learning new tasks.”
Meta-Learning Probabilistic Inference for Prediction (2018, methodology): “Standard application of amortized VI (Kingma and Welling, 2014; Rezende et al., 2014; Kingma et al., 2015; Blundell et al., 2015) for ψ in the multi-task discriminative model optimizes the Monte Carlo approximated free-energy w.r.t. φ and θ : where ψ ( t ) l ∼ q φ ( ψ D ( t ) , θ ) .”
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks (2015, methodology): “Recent results at the intersection of Bayesian research and deep learning offer interpretation of common deep learning techniques through Bayesian eyes [11–16].”
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning (2015, methodology): “These have been used to obtain new approximations for Bayesian neural networks that perform as well as dropout [26].”
A General Framework for Uncertainty Estimation in Deep Learning (2019, neutral): “, exponential-family [2], [5], [8], [9].”
Deep Variational Information Bottleneck (2017, methodology): “4 Even if our choice of encoding distribution and variational prior do not admit an analytic KL, we could similarly reparameterize through a sample of the divergence (Kingma & Welling, 2014; Blundell et al., 2015).”
Decentralized Bayesian Learning over Graphs (2019, neutral): “In particular, our learning rule also generalizes various Bayesian inference techniques such as [9–11, 5] and variational continual learning techniques such as [9, 11].”
Peer-to-peer Federated Learning on Graphs (2019, methodology): “We use the Bayes by Backprop training algorithm proposed by Blundell et al. (2015) to learn the Gaussian variational posterior.”
Variational Continual Learning (2017, methodology): “We merge online variational inference (VI) (Ghahramani and Attias, 2000; Sato, 2001; Broderick et al., 2013) with Monte Carlo VI for neural networks (Blundell et al., 2015) to yield variational continual learning (VCL).”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, background): “While the diagonal covariance approximation is standard in Bayesian deep learning [6, 34], it can be too restrictive.”
Overcoming Catastrophic Forgetting by Incremental Moment Matching (2017, background): “IMM approximates the mixture of Gaussian posterior with each component representing parameters for a single task to one Gaussian distribution for a combined task.”
S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, background): “Stochastic Neural Networks: Traditional neural networks cannot model uncertainty well due to their deterministic nature [5].”
Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, methodology): “Third, along with a standard smoothing procedure, [8] averages the prediction using 10 samples of the weights.”
Posterior Meta-Replay for Continual Learning (2021, methodology): “Here, we employ several approximations to the posterior based on variational inference [4] from prior work, ranging from simple and scalable methods with a mean-field variational family like Bayes-by-Backprop (BbB, [5]) to methods with complex but rich variational families like the spectral Stein gr”
Routing Networks with Co-training for Continual Learning (2020, methodology): “The uncertainty of each model weight is estimated using the Bayes-by-backprop (BBB) method (Blundell et al., 2015).”
Continual Deep Learning by Functional Regularisation of Memorable Past (2020, methodology): “Poor performance of VCL is most likely due to the difficulty of using Bayes By Backprop [7] on CNNs3 [23, 31].”
Continual Learning by Asymmetric Loss Approximation With Single-Side Overestimation (2019, background): “Variational continual learning (VCL) [27] combines the online variational inference [9, 31, 5] with Monte Carlo sampling [4] using neural networks, but it requires a relatively large amount of computational cost to infer an approximate posterior distribution.”
Uncertainty-guided Continual Learning with Bayesian Neural Networks (2019, methodology): “2 can be approximated using N Monte Carlo samples from the variational posterior [4]:”