Probable networks and plausible predictions - a review of practical Bayesian methods for supervised neural networks

By David John Cameron MacKay. 1995

Subfields: Model Architectures, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Probabilistic & Bayesian Methods

Citations in the corpus

A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, background): “Denker and LeCun [7], and MacKay [44] investigated the posterior probability distributions of neural networks and treated the search in the model space (the space of architectures, weight decay, regularizers, etc..) as an inference problem and tried to solve it using Laplace approximations.”
Weight Uncertainty in Neural Networks (2015, methodology): “In this work, we introduce an efficient, principled algorithm for regularisation built upon Bayesian inference on the weights of the network (MacKay, 1992; Buntine and Weigend, 1991; MacKay, 1995).”
Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks (2020, positive): “Letting d := ∇fθ(x)|µ, we do a first-order Taylor expansion of fθ at µ (MacKay, 1995): fθ(x) ≈ fµ(x) + d>(θ − µ).”
Machine Theory of Mind (2018, neutral): “There should be nothing surprising about seeing the ToM-net learn to approximate Bayes-optimal online inference; this should be expected given more general results about inference and meta-learning with neural networks (MacKay, 1995; Finn & Levine, 2017).”
Bayesian Compression for Deep Learning (2017, background): “From the Bayesian viewpoint, similar pruning schemes have been explored at [47, 53, 39, 34].”
Task Clustering and Gating for Bayesian Multitask Learning (2003, methodology): “In many applications, this assumption is a little too simplistic and we have to consider more sophisticated priors.”
Bayesian Gaussian process models : PAC-Bayesian generalisation error bounds and sparse approximations (2003, methodology): “B.1 Extended PAC-Bayesian Theorem: an Example . . . . . . . . . . 197 B.2 Details of Proof of Theorem 3.2 . . . . . . . . . . . . . . . . . . . 199 B.3 Proof of Theorem 3.3 . . . . . . . . . . . . . . . . . . . . . . . . . 199 B.4 Efficient Evaluation of the Laplace GP Gibbs Classifier . . . . . .”
Advanced Lectures on Machine Learning (2004, background): “The decrease in the marginal likelihood as additional parameters are added has been called the automatic Occam’s Razor [59, 60, 61].”
Towards dropout training for convolutional neural networks (2015, background): “Other common forms of regularization include early stopping, Bayesian fitting (Mackay, 1995), weight elimination (Ledoux & Talagrand, 1991) and data augmentation.”
Variational free energy and the Laplace approximation (2007, background): “This enables the inversion to infer which parameters are relevant and which are not (MacKay, 1995a,b).”
Does feature selection improve classification accuracy? Impact of sample size and feature selection on classification using anatomical magnetic resonance images (2012, background): “These include lasso regression (Tibshirani, 1996), elastic net (Zou and Hastie, 2005), automatic relevance determination (Chu et al., 2010; MacKay, 1995; Tipping, 2001), and other sparse models (Carroll et al., 2009; Grosenick et al., 2008; Ryali et al., 2010; Yamashita et al., 2008).”
Bayesian Framework for Least-Squares Support Vector Machine Classifiers, Gaussian Processes, and Kernel Fisher Discriminant Analysis (2002, methodology): “The differences with the expressions obtained in (MacKay, 1995) are due to the fact that no regularization is applied on the bias term b and that all practical expressions are obtained in the dual space by applying the Mercer condition.”
Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference (2018, methodology): “to account and utilizing a minimum description length (MDL) loss. This penalises non-robust weights by means of an approximate penalty based upon perturbations of the weights on the outputs. [15] and [16] investigated the posterior probability distributions of neural networks by using Laplace approx”
The Priority Heuristic: Making Choices Without Trade-Offs (2006, background): “Models with smaller numbers of adjustable parameters, which embody Occam’s razor, have a higher posterior probability in Bayesian model comparison (MacKay, 1995; Roberts & Pashler, 2000).”