SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient

By Aaron Mishkin, Frederik Kunstner, Didrik Nielsen, Mark W. Schmidt, Mohammad Emtiyaz Khan. 2018

67 citations recorded by the source corpus.

Uncertainty estimation in large deep-learning models is a computationally challenging task, where it is difficult to form even a Gaussian approximation to the posterior distribution. In such situations, existing methods usually resort to a diagonal approximation of the covariance matrix despite the fact that these matrices are known to give poor uncertainty estimates. To address this issue, we propose a new stochastic, low-rank, approximate natural-gradient (SLANG) method for variational inference in large deep models. Our method estimates a “diagonal plus low-rank” structure based solely on back-propagated gradients of the network log-likelihood. This requires strictly less gradient computations than methods that compute the gradient of the whole variational objective. Empirical evaluations on standard benchmarks confirm that SLANG enables faster and more accurate estimation of uncertainty than mean-field methods, and performs comparably to state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Lifelong Generative Modeling (2017, background): “requirement for the parameters of a neural network and there is active research in Bayesian neural networks that attempts to relax this constraint [74, 75, 80].”
A survey of uncertainty in deep neural networks (2021, background): “In order to make more expressive variational distributions feasible for deep neural networks, several works proposed to infer using the matrix normal distribution [144], [145], [146] or more expressive variants [147], [148] where the covariance matrix is decomposed into the”
Uncertainty Estimation with Infinitesimal Jackknife, Its Distribution and Mean-Field Approximation (2020, background): “To this end, many approaches assume layer-wise independence [41] and low-rank components [33, 38] and in some cases, restrict uncertainty quantiﬁcation to only a few layers [48] – in particular, the last layer [26].”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, unknown): “[368] designed a novel stochastic, low-rank, approximate natural-gradient (SLANG) technique for VI in huge deep models to tackle this issue.”
Limitations of the Empirical Fisher Approximation (2019, methodology): “The EF has been used in the context of variational inference by various authors [Graves, 2011, Zhang et al., 2018, Salas et al., 2018, Khan et al., 2018, Mishkin et al., 2018], some of which have drawn further connections between NGD and Adam.”
Eigenvalue Corrected Noisy Natural Gradient (2018, unknown): “Concurrently, Mishkin et al. [2018] adopted a "diagonal plus low-rank" approximation.”