A Kronecker-factored approximate Fisher matrix for convolution layers

By Roger Baker Grosse, James Martens. 2016

317 citations recorded by the source corpus.

Second-order optimization methods such as natural gradient descent have the potential to speed up training of neural networks by correcting for the curvature of the loss function. Unfortunately, the exact natural gradient is impractical to compute for large models, and most approximations either require an expensive iterative procedure or make crude approximations to the curvature. We present Kronecker Factors for Convolution (KFC), a tractable approximation to the Fisher matrix for convolutional networks based on a structured probabilistic model for the distribution over backpropagated derivatives. Similarly to the recently proposed Kronecker-Factored Approximate Curvature (K-FAC), each block of the approximate Fisher matrix decomposes as the Kronecker product of small matrices, allowing for efficient inversion. KFC captures important curvature information while still yielding comparably efficient updates to stochastic gradient descent (SGD). We show that the updates are invariant to commonly used reparameterizations, such as centering of the activations. In our experiments, approximate natural gradient descent with KFC was able to train convolutional networks several times faster than carefully tuned SGD. Furthermore, it was able to train the networks in 10-20 times fewer iterations than SGD, suggesting its potential applicability in a distributed setting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “Other optimizers, such as KFAC (Martens and Grosse, 2015; Grosse and Martens, 2016; Ba et al., 2017), or optimization techniques designed speciﬁcally for massively data parallel systems (e.g. Li et al., 2014), might allow perfect scaling to extend much further.”
Critical Learning Periods in Deep Neural Networks (2017, methodology): “We also considered computing the log-determinant of the matrix using the Kronecker-Factorized approximation (40, 41) in order to capture the behavior of off-diagonal terms, but we observed the same qualitative trend as the trace.”
DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, methodology): “K-FAC [12,30] proposed to approximately factorize the block diagonals of a neural network’s Fisher information matrix into a Kronecker product of covariance matrices:”
Continual Learning With Extended Kronecker-Factored Approximate Curvature (2020, methodology): “, the Hessian of a 1024×1024 fully connected layer takes 4 TB in memory with single-precision floating-point numbers), the K-FAC method factors the Hessian into two relatively small matrices [4, 10, 23, 24] so that Equation 8 is approximated by”
Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation (2017, background): “Grosse and Martens [7] have recently extended the K-FAC algorithm to handle convolutional networks.”
Meta-Curvature (2019, neutral): “Kronecker-factored Approximate Curvature (K-FAC) [24, 14] approximates the Fisher matrix by the Kronecker product, e.”
Noisy Natural Gradient as Variational Inference (2017, methodology): “Following on that work, K-FAC has been adopted in many tasks (Grosse & Martens, 2016; Wu et al., 2017) to gain optimization beneﬁts, and was shown to be amenable to distributed computation (Ba et al., 2017).”
Exact natural gradient in deep linear networks and its application to the nonlinear case (2018, methodology): “…for practical applications, previous studies suggested a block-diagonal approximation to the inverse Fisher information matrix, in the nonlinear case (K-FAC, [Martens and Grosse, 2015; Grosse and Martens, 2016; Ba et al., 2016], see also [Heskes, 2000; Povey et al., 2014; Desjardins et al., 2015]).”
Optimization and Generalization of Regularization-Based Continual Learning: a Loss Approximation Viewpoint. (2020, methodology): “(2018) is similar to that of EWC, with the key difference being that it uses Kronecker product approximation of the Fisher information matrix (Martens and Grosse, 2015; Grosse and Martens, 2016).”
A Scalable Laplace Approximation for Neural Networks (2018, unknown): “As demonstrated in (Grosse & Martens, 2016), Kronecker factored curvature methods are applicable to convolutional layers by interpreting them as matrix-matrix multiplications.”
Neuroevolution of self-interpretable agents (2020, background): “Matrix factorization applied to weights has been proposed in the deep learning literature [33, 80], and are also present in recommender systems [55] to represent relationships between different inputs.”
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, unknown): “Earlier work called Layer-wise OBS (L-OBS) [6] was inspired by the K-FAC approximation [29, 12]: L-OBS approximates the OBS framework not for the global objective, but for a quadratic per-layer loss, while also pruning all weights based on a single Hessian computation.”
Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods (2018, methodology): “…that understanding optimization as inference, taking into account the dynamics in the other weights as they are optimized, will allow for the development of improved optimizers, for instance by exploiting Kronecker factorisation (Martens & Grosse, 2015; Grosse & Martens, 2016; Zhang et al., 2017).”
Adaptive norms for deep learning with regularized Newton methods (2019, result): “Nevertheless, the above works report promising preliminary results, most notably Grosse & Martens (2016) ﬁnd that K-FAC can be faster than SGD on a small convnet.”
M-FAC: Efficient Matrix-Free Approximations of Second-Order Information (2021, background): “Another relative drawback is that the Kronecker factorization only occurs naturally for fully-connected layers; but there is work on extensions to other layer types [11, 30], via additional approximations.”