Limitations of the Empirical Fisher Approximation

By Frederik Kunstner, Philipp Hennig, Lukas Balles. 2019

290 citations recorded by the source corpus.

Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information. Several highly visible works have advocated an approximation known as the empirical Fisher, drawing connections between approximate second-order methods and heuristics like Adam. We dispute this argument by showing that the empirical Fisher---unlike the Fisher---does not generally capture second-order information. We further argue that the conditions under which the empirical Fisher approaches the Fisher (and the Hessian) are unlikely to be met in practice, and that, even on simple optimization problems, the pathologies of the empirical Fisher can have undesirable effects.

Subfields: Artificial Intelligence, Optimization & Training, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Statistical Learning Theory

Citations in the corpus

Training Neural Networks with Fixed Sparse Masks (2021, methodology): “It has been shown that using the empirical Fisher can lead to degenerate behavior when used as a preconditioner in an optimizer [34, 26].”
Natural continual learning: success is a journey, not (just) a destination (2021, methodology): “…terms of the change in probability distributions It can be shown that the direction of steepest decent for small step sizes is in this case given by (Kunstner et al., 2019; Amari, 1998) where F is the Fisher information matrix We thus get an update rule of the form which has units of [ θ ] and…”
Provable Continual Learning via Sketched Jacobian Approximations (2021, methodology): “Specifically, it corresponds to “online” EWC where instead of the original Fisher matrix, the empirical Fisher matrix is used (see (Kunstner et al., 2019) on the relations of the original and empirical Fisher matrices).”
Laplace Redux - Effortless Bayesian Deep Learning (2021, methodology): “3If, instead of taking expectation in (8), we use the training label yn, we call the matrix the empirical Fisher, which is distinct from the Fisher [38, 40].”
M-FAC: Efficient Matrix-Free Approximations of Second-Order Information (2021, methodology): “[21] performed an in-depth analysis of the empirical Fisher, making the point that, in theory, the approximation could become meaningless if all sample gradients are zero.”
BackPACK: Packing more into backprop (2019, methodology): “For popular loss functions, it coincides with the Fisher information matrix used in natural gradient methods (Amari, 1998); for a more in depth discussion of the equivalence, see the reviews of Martens (2014) and Kunstner et al. (2019).”
WoodFisher: Efficient second-order approximations for model compression (2020, background): “(1.11) We refer the readers to Kunstner et al. (2019) for this proof and a nice exposition about the topic.”
Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networks (2021, methodology): “To address this, we note that several empirical analyses of neural network curvature have found that the Hessian and dataset Fisher FDw∗ exhibit rapid spectral decay [Sagun et al., 2017, Madras et al., 2020], meaning that there are only a small number of significant eigenvalues and eigenvectors.”
The power of quantum neural networks (2020, methodology): “, limk→∞ F̃k(θ) = F (θ) [33].”
Lightlike Neuromanifolds, Occam's Razor and Deep Learning (2019, unknown): “a recent discussion (Kunstner et al., 2019).”
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (2023, unknown): “Across different total number of steps, Sophia outperforms AdamW and Lion with a large margin as the main experiments we presented in Section 3.2.”
Information Geometry of Orthogonal Initializations and Training (2018, methodology): “estimating the Lipschitz constant is NP-hard (Kunstner et al., 2019), therefore we seek to find local measures of curvature along the optimization trajectory.”
WoodFisher: Efficient Second-Order Approximation for Neural Network Compression (2020, methodology): “Second-order information in the form of Inverse-Hessian Vector Products (IHVP) has several uses in optimization and machine learning [29, 30].”