Bayesian learning for neural networks

By Radford M. Neal. 1995

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Improving neural networks by preventing co-adaptation of feature detectors (2012, methodology): “For complicated model classes, like feedforward neural networks, Bayesian methods typically use a Markov chain Monte Carlo method to sample models from the posterior distribution [13].”
Scaling description of generalization with number of parameters in deep learning (2019, background): “At initialization t = 0, fN,t=0 is a random function whose limiting distribution as N → ∞ is an explicit Gaussian [46, 47, 48].”
Neural Tangent Kernel: Convergence and Generalization in Neural Networks (2018, methodology): “At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit [12, 9], thus connecting them to kernel methods.”
Natural Language Processing (Almost) from Scratch (2011, background): “On the other hand, multiple training times could be improved using smart sampling strategies (Neal, 1996).”
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks (2015, background): “(1) One way to define a distribution over a parametric set of functions is to place a prior distribution over a neural network’s weights, resulting in a Bayesian NN [21, 22].”
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning (2015, background): “n deep reinforcement learning [14]. 2Related Research It has long been known that inﬁnitely wide (single hidden layer) MLPs with distributions placed over their weights converge to Gaussian processes [15, 16]. This known relation is through a limit argument that does not allow the transfer of proper”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, methodology): “y. Since exact Bayesian inference is computationally intractable for NNs, a variety of approximations have been developed including Laplace approximation [40], Markov chain Monte Carlo (MCMC) methods [46], as well as recent work on variational Bayesian methods [6, 19, 39], assumed density ﬁltering [”
A General Framework for Uncertainty Estimation in Deep Learning (2019, background): “ections. A. Dropout to Estimate Model Uncertainty To account for model uncertainty in deep learning, a distribution is placed over neural network (NN) weights !, deﬁning a Bayesian neural network [7]–[9]. The work of Gal et al. [10] provides a mathematically grounded framework to capture model uncer”
Variational Inference: A Review for Statisticians (2016, background): “Automatic relevance determination (ARD) assigns a separate prior for each regression coefficient; the idea is to automatically shrink irrelevant coefficients during inference (MacKay, 1992; Neal, 2012; Tipping, 2001; Wipf and Nagarajan, 2008).”
Wireless Network Intelligence at the Edge (2018, background): “(a) Gaussian Process (GP): Gaussian-distributed priors with infinitely many perceptrons in a single hidden layer result in GP via the central limit theorem [127].”
Scalable Bayesian Optimization Using Deep Neural Networks (2015, methodology): “We integrate out α and β using slice sampling (Neal, 2000) according to the methodology of Snoek et al. (2012) over the marginal likelihood, which is given by”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, methodology): “Markov chain Monte Carlo (MCMC) was at one time a gold standard for inference with neural networks, through the Hamiltonian Monte Carlo (HMC) work of Neal [49].”
Deep learning in neural networks: An overview (2014, background): “n and van Camp, 1993). In the NN case, the MDL principle suggests that low NN weight complexity corresponds to high NN probability in the Bayesian view (e.g., MacKay, 1992; Buntine and Weigend, 1991; Neal, 1995; De Freitas, 2003), and to high generalization performance (e.g., Baum and Haussler, 1989”
Adversarial Vulnerability of Neural Networks Increases With Input Dimension (2018, methodology): “ssumptions have been widely used to analyze neural nets and are at the heart of very inﬂuential and successful prior work (e.g., equivalence between neural nets and Gaussian processes as pioneered by Neal 1996). Nevertheless, they do not hold after training. That is why all our statements in this se”
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning (2018, background): “nty estimates. Unfortunately, Bayesian inference remains computationally hard for neural networks. Hence, different degrees of approximations are made to reduce the computational overhead [40], [41], [42], [43], including about the prior that is speciﬁed for the parameters of the neural network. Des”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “, the Laplace approximation (MacKay, 1992b), Hamiltonian Monte Carlo (Neal, 1995), expectation propagation (Jylänki et al.”
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, background): “he cubic complexity of Gaussian processes to become the limiting factor in many practical applications. We therefore plan to extend this work to other model classes, such as Bayesian neural networks (Neal, 1996; Hernández-Lobato and Adams, 2015; Blundell et al., 2015; Springenberg et al., 2016; Klei”