A Scalable Laplace Approximation for Neural Networks

By Hippolyt Ritter, Aleksandar Botev, David Barber. 2018

513 citations recorded by the source corpus.

Subfields: Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, methodology): “…Laplace approximation has a long history in GP and NN literature (Tierney et al., 1989; Denker and LeCun, 1991; Rasmussen and Williams, 2006; MacKay, 1992; Ritter et al., 2018), and the theoretical connection between a Laplace-approximated DNN and GP has being explored recently (Khan et al., 2019).”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, methodology): “In particular SWAG achieves higher test likelihood compared to many state-of-the-art approaches, including MC-Dropout [14], temperature scaling [19], SGLD [59], KFAC-Laplace [54] and SWA [27] on CIFAR-10, CIFAR-100 and ImageNet, on a range of architectures.”
Posterior Meta-Replay for Continual Learning (2021, background): “We investigate how this problem can be tackled by selecting the task-embedding that yields the lowest predictive uncertainty, effectively resorting to out-of-distribution (OOD) detection (Louizos & Welling, 2017; Ritter et al., 2018).”
Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting (2018, methodology): “Our work links the Kronecker factored Laplace approximation [34] to Bayesian online learning [31] similar to how Variational Continual Learning [30] connects Online Variational Learning [7, 13] to Bayes-by-Backprop [1].”
Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks (2020, methodology): “LLLA with a matrix normal distribution as described in the previous section is a special case of KFLA (Ritter et al., 2018).”
Practical Deep Learning with Bayesian Principles (2019, background): “Many other approaches have recently been proposed to compute posterior approximations by training deterministic networks [46, 37, 38].”
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (2018, methodology): “The goal of this approach is similar to other approaches that employ structured approximations (Ritter et al., 2018; Louizos & Welling, 2016; Sun et al., 2017).”
Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, background): “Recent work has largely focused on scalable inference (e.g., Blundell et al., 2015; Gal & Ghahramani, 2016; Kendall & Gal, 2017; Ritter et al., 2018; Khan et al., 2018; Maddox et al., 2019), function-space inspired priors (e.”
SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient (2018, methodology): “Another recent work has applied K-FAC to find a Laplace approximation [29].”
Laplace Redux - Effortless Bayesian Deep Learning (2021, methodology): “LAs are widely used to approximate the posterior distribution in logistic regression [31], Gaussian process classification [32, 33], and also for Bayesian neural networks (BNNs), both shallow [34] and deep [23].”
Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning (2020, methodology): “al inference (Blundell et al., 2015; Kingma et al., 2015; Louizos & Welling, 2017), batch normalization (Ioffe & Szegedy, 2015; Teye et al., 2018; Atanov et al., 2019), Laplace approximation (Ritter et al., 2018) and many more. Snapshot-based methods aim to obtain sets of weights for deep le”
Subspace Inference for Bayesian Deep Learning (2019, methodology): “Subspace inference is competitive with SWAG and consistently outperforms most of the other baselines, including MC-dropout (Gal and Ghahramani, 2016), temperature scaling (Guo et al., 2017) and KFAC-Laplace Ritter et al. (2018).”
A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving (2020, methodology): “Multiple techniques for generating approximate solutions of the posterior distribution have been proposed [25]–[34].”
Uncertainty Estimation with Infinitesimal Jackknife, Its Distribution and Mean-Field Approximation (2020, methodology): “For BNN ( KFAC ) and RUE , we use mini-batch approximations on subsets of the training set to scale up on ImageNet, as suggested in [41, 42].”
Fast Uncertainty Estimates and Bayesian Model Averaging of DNNs (2018, methodology): “Next, we compare the SWAG methods with diagonal Laplace approximations on a toy regression problem: modelling y ∼ x(3) + N(0, 3(2)) with a simple two-layer MLP as in [16].”
A survey of uncertainty in deep neural networks (2021, methodology): “1[20] 3[30] 4[149] 5[75] 6[213] 7[129] 8[63] 9[80] 10[150] 11[143] 12[84] 13[235] 14[145] 15[135] 16[146] 17[272]”
Model Selection in Bayesian Neural Networks via Horseshoe Priors (2017, background): “, 2014), probabilistic backpropagation (HernándezLobato and Adams, 2015) as well as scalable Laplace approximations (Ritter et al., 2018) have reinvigorated interest in BNNs by allowing efficient scalable inference.”
Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networks (2021, methodology): “First, we compare with the two closely-related approaches, namely, Local Ensembles [Madras et al., 2020] and KFAC Laplace approximation [Ritter et al., 2018], which use curvature estimation to augment a trained model with uncertainty estimates.”
Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning (2019, background): “…prior-focused Bayesian approaches to continual learning in which the posterior at the end of learning a task becomes a prior when learning the next task [Kirk-patrick et al., 2017, Zenke et al., 2017, Chaudhry et al., 2018, Nguyen et al., 2018, Farquhar and Gal, 2018b, Ritter et al., 2018].”