Practical Gauss-Newton Optimisation for Deep Learning

By Aleksandar Botev, Hippolyt Ritter, David Barber. 2017

266 citations recorded by the source corpus.

We present an efficient block-diagonal ap- proximation to the Gauss-Newton matrix for feedforward neural networks. Our result- ing algorithm is competitive against state- of-the-art first order optimisation methods, with sometimes significant improvement in optimisation performance. Unlike first-order methods, for which hyperparameter tuning of the optimisation parameters is often a labo- rious process, our approach can provide good performance even when used with default set- tings. A side result of our work is that for piecewise linear transfer functions, the net- work objective function can have no differ- entiable local maxima, which may partially explain why such transfer functions facilitate effective optimisation.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Signal Processing

Citations in the corpus

BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “…of weight matrix has been widely used in second-order optimization techniques to speed up the training process (LeCun et al., 1990; Amari, 1998), it still remains infeasible to calculate explicit Hessian directly due to the intensive computation burden (Martens & Grosse, 2015; Botev et al., 2017).”
Continual Learning With Extended Kronecker-Factored Approximate Curvature (2020, methodology): “then Equation 9, K-FAC, can be written in a simple block matrix form [4, 24] as”
Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting (2018, methodology): “Martens and Grosse [23] and Botev et al. [2] both develop block-diagonal Kronecker factored approximations to the Fisher and Gauss-Newton matrix of fully connected neural networks respectively, which in turn both are positive semi-deﬁnite approximations of the Hessian.”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “Botev et al. (2017) observed that for networks with piece-wise linear activation functions, such as the popular RELUs (given by [φi(si)]j = max([si]j , 0)), the GGN and the Hessian will coincide on the diagonal blocks whenever the latter is well-defined.”
DAGs with NO TEARS: Continuous Optimization for Structure Learning (2018, methodology): “…deep neural networks are often learned using various descendants of stochastic gradient descent (SGD) (Bousquet and Bottou, 2008; Kingma and Ba, 2014; Bottou et al., 2016), although recent work has proposed other techniques such as ADMM (Taylor et al., 2016) and Gauss-Newton (Botev et al., 2017).”
Up or Down? Adaptive Rounding for Post-Training Quantization (2020, methodology): “Writing this in matrix formulation (for flattened w), we have (Botev et al., 2017)”
Limitations of the Empirical Fisher Approximation (2019, methodology): “The generalized Gauss-Newton [Schraudolph, 2002] and natural gradient descent [Amari, 1998] methods have inspired a line of work on approximate second-order optimization [Martens, 2010, Botev et al., 2017, Park et al., 2000, Pascanu and Bengio, 2014, Ollivier, 2015].”
Laplace Redux - Effortless Bayesian Deep Learning (2021, background): “[20] Aleksandar Botev, Hippolyt Ritter, and David Barber.”
DeepOBS: A Deep Learning Optimizer Benchmark Suite (2019, background): “, 2018), incorporation of second-order information (Martens, 2010; Martens & Grosse, 2015; Botev et al., 2017; Zhang et al., 2017; Chen et al., 2018), and automating optimization (Schaul et al.”
Revisiting Recursive Least Squares for Training Deep Neural Networks (2021, methodology): “The second type, such as the K-FAC algorithm [19] and the K-FRA algorithm [20], uses the block-diagonal approximation of the Hessian to reduce the computational cost.”
Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks (2019, methodology): “It is also closely connected to Gauss-Newton optimization, suggesting it should achieve fast convergence in certain settings [Pascanu and Bengio, 2013, Martens, 2014, Botev et al., 2017].”
When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations (2021, methodology): “We can gain insights into the above ﬁndings by the recursive formulation of Hessian matrices for MLPs (Botev et al., 2017).”
A Scalable Laplace Approximation for Neural Networks (2018, methodology): “The diagonal blocks of the Hessian and Gauss-Newton matrix are equal for neural networks with piecewise linear activation functions (Botev et al., 2017), thus both methods can be used to directly approximate the diagonal blocks of the Hessian of such networks, as the Gauss-Newton and Fisher are…”
Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks (2018, background): “In the training of deep neural networks, FIM can be assumed as the curvature matrix in the parameter space [4, 6, 21].”
BackPACK: Packing more into backprop (2019, methodology): “– Diagonal of the GGN and the Hessian – KFAC (Martens & Grosse, 2015) – KFRA and KFLR (Botev et al., 2017) These quantities are only deﬁned, or reasonable to compute, for a subset of models: The concept of individual gradients for each sample in a mini-batch or the estimate of the variance requires…”
Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networks (2021, background): “An alternative is to consider the Fisher (or the Gauss–Newton [Botev et al., 2017]) approximation of the full Hessian.”
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (2023, unknown): “…than the ﬁrst term (Sankar et al., 2021), which is often referred to as the Gauss-Newton matrix (Dennis Jr & Schnabel, 1996; Ortega & Rheinboldt, 2000; Schraudolph, 2002; Chen, 2011) and used as pre-conditioners in second-order optimizers (Botev et al., 2017; Martens, 2020; Gargiani et al., 2020).”
BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction (2021, methodology): “GN matrix can be written in matrix form (Botev et al., 2017) as H(θ) ≈ G(θ) = Jz(n)(θ)TH(z (n))Jz(n)(θ), (6) where Jz(n)(θ) is the Jacobian matrix of the network output with respect to the network parameters.”
Information Geometry of Orthogonal Initializations and Training (2018, methodology): “In this context, the Fisher information matrix is deﬁned as [2, 7, 22], which can be expanding using the chain rule: Each block of the Fisher information matrix with respect to parameters a, b ∈ Θ can further be expressed as where the ﬁnal layer Hessian H g is deﬁned as ∇ 2 h g log p θ ( y x 0 ) .…”
Second-order Optimization Method for Large Mini-batch: Training ResNet-50 on ImageNet in 35 Epochs (2018, background): “In the training of deep neural networks, FIM can be assumed as the curvature matrix in the parameter space [4, 6, 20].”