Eigenvalue Corrected Noisy Natural Gradient
By Juhan Bae, Guodong Zhang, Roger Baker Grosse. 2018
19 citations recorded by the source corpus.
Variational Bayesian neural networks combine the flexibility of deep learning with Bayesian uncertainty estimation. However, inference procedures for flexible variational posteriors are computationally expensive. A recently proposed method, noisy natural gradient, is a surprisingly simple method to fit expressive posteriors by adding weight noise to regular natural gradient updates. Noisy K-FAC is an instance of noisy natural gradient that fits a matrix-variate Gaussian posterior with minor changes to ordinary K-FAC. Nevertheless, a matrix-variate Gaussian posterior does not capture an accurate diagonal variance. In this work, we extend on noisy K-FAC to obtain a more flexible posterior distribution called eigenvalue corrected matrix-variate Gaussian. The proposed method computes the full diagonal re-scaling factor in Kronecker-factored eigenbasis. Empirically, our approach consistently outperforms existing algorithms (e.g., noisy K-FAC) on regression and classification tasks.
Subfields: Artificial Intelligence, Optimization & Training, Probabilistic & Bayesian Methods
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Guodong Zhang
- Roger Baker Grosse
- Chaoqi Wang
- Sanja Fidler
- Jiaxin Shi
- Rudolph Triebel
- Xiaoxiang Zhu
- Ribana Roscher
In Optimization & Training
- Guodong Zhang
- Chaoqi Wang
- Sanja Fidler
- Roger Baker Grosse
- Rudolph Triebel
- Muhammad Shahzad
- Jiaxin Shi
- Xiaoxiang Zhu
In Probabilistic & Bayesian Methods
- Chaoqi Wang
- Roger Baker Grosse
- Guodong Zhang
- Sanja Fidler
- Jiaxin Shi
- Muhammad Shahzad
- Cedrique Rovile Njieutcheu Tassi
- Richard Bamler
Citations in the corpus
- A survey of uncertainty in deep neural networks (2021, background): “In order to make more expressive variational distributions feasible for deep neural networks, several works proposed to infer using the matrix normal distribution [144], [145], [146] or more expressive variants [147], [148] where the covariance matrix is decomposed into the”
- EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis (2019, methodology): “Specifically, we consider the Kronecker-factored eigenbasis (KFE) (George et al., 2018; Bae et al., 2018), in which the Hessian for a given layer is closer to diagonal.”
- Functional Variational Bayesian Neural Networks (2019, background): “There has also been much work (Louizos & Welling, 2016; Sun et al., 2017; Zhang et al., 2018; Bae et al., 2018) on modelling the correlations between weights using more complex Gaussian variational posteriors.”
- A Trace-restricted Kronecker-Factored Approximation to Natural Gradient (2020, unknown): “…considered the efficient Kronecker-factored approximation to the FIM, such as KFAC (Martens and Grosse, 2015; Grosse and Martens, 2016; Martens et al., 2018,Martens and Grosse, 2015; Zhang et al., 2018; Bae et al., 2018) and Eigenvalue-corrected Kronecker Factorization (EKFAC) (George et al., 2018).”