A Practical Bayesian Framework for Backpropagation Networks

By David John Cameron MacKay. 1992

Subfields: Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning (2015, background): “ow us to translate properties from the Gaussian process to ﬁnite NNs easily. Finite NNs with distributions placed over the weights have been studied extensively as Bayesian neural networks (Neal,1995;MacKay, 1992). These offer robustness to over-ﬁtting as well, but with challenging inference and add”
Collaborative Deep Learning for Recommender Systems (2014, background): “If we assume that both the clean input Xc and the corrupted input X0 are observed, similar to [4, 19, 3, 7], we can define the following generative process:”
Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, methodology): “…DNN Laplace approximation has a long history in GP and NN literature (Tierney et al., 1989; Denker and LeCun, 1991; Rasmussen and Williams, 2006; MacKay, 1992; Ritter et al., 2018), and the theoretical connection between a Laplace-approximated DNN and GP has being explored recently (Khan et…”
A General Framework for Uncertainty Estimation in Deep Learning (2019, background): “Similarly to Bayesian approaches [25], [26], [27], [28], [3], we represent this uncertainty by placing a distribution over the neural network weights, ω .”
Scalable Bayesian Optimization Using Deep Neural Networks (2015, methodology): “The idea of applying Bayesian methods to neural networks has a rich history in machine learning (MacKay, 1992; Hinton & van Camp, 1993; Buntine & Weigend, 1991; Neal, 1995; De Freitas, 2003) and Mnih & Gregor (2014), where a neural network is used in a variational approximation to the posterior…”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, methodology): “In the late 1990s, Bayesian methods were the state-of-the-art approach to learning with neural networks, through the seminal works of Neal [49] and MacKay [40].”
Deep learning in neural networks: An overview (2014, methodology): “Often weight priors are implicit in additional penalty terms (MacKay, 1992) or in methods based on validation sets (Craven & Wahba, 1979; Eubank, 1988; Golub, Heath, & Wahba, 1979; Hastie & Tibshirani, 1990; Mosteller & Tukey, 1968; Stone, 1974), Akaike’s information criterion and final prediction…”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “In principle, the Bayesian approach to learn neural networks does not have problems of tuning a large amount of hyperparameters or overfitting the training data (MacKay, 1992b;a; Neal, 1995; HernándezLobato & Adams, 2015).”
Overcoming Multi-Model Forgetting (2019, positive): “By relying on Laplace approximations in neural networks (MacKay, 1992) and the connection between the Fisher information matrix and second-order derivatives (Pascanu & Bengio, 2014), EWC is then formulated as the loss L L P − , where A and B refer to two different tasks , θ encodes the network…”
An overview of deep learning in medical imaging focusing on MRI (2018, background): “The ideas date back to the early 90s [354, 355, 356], but the field has recently seen renewed interest from the machine learning community at large, as new ways of computing uncertainty estimates from state of the art deep learning models have been developed [59, 65, 357].”
Getting a CLUE: A Method for Explaining Uncertainty Estimates (2020, background): “There is growing interest in probabilistic machine learning models, which aim to provide reliable estimates of uncertainty about their predictions (MacKay, 1992).”
Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data (2019, methodology): “Uncertainty quantification for DNNs is often studied under the re-emerging framework of Bayesian deep learning(1) [15], mostly using variational inference for approximate posterior of model parameters, e.”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, background): “Constructing good Bayesian NN models remains an open problem [35], [10], [36], [37], although recent promising work”
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes (2018, neutral): “The Laplace approximation (Laplace, 1986; MacKay, 1992a;b) is applicable in this case as it replaces a point estimate of an integral with the volume of a Gaussian centered at a mode of the integrand, thereby forming a local quadratic approximation.”
Probabilistic Model-Agnostic Meta-Learning (2018, background): “Bayesian neural networks [27, 18, 29, 1] have been studied extensively as a way to incorporate uncertainty into deep networks.”