Universal statistics of Fisher information in deep neural networks: mean field approach

By Ryo Karakida, Shotaro Akaho, Shun-ichi Amari. 2018

187 citations recorded by the source corpus.

The Fisher information matrix (FIM) is a fundamental quantity to represent the characteristics of a stochastic model, including deep neural networks (DNNs). The present study reveals novel statistics of FIM that are universal among a wide class of DNNs. To this end, we use random weights and large width limits, which enables us to utilize mean field theories. We investigate the asymptotic statistics of the FIM’s eigenvalues and reveal that most of them are close to zero while the maximum eigenvalue takes a huge value. Because the landscape of the parameter space is defined by the FIM, it is locally flat in most dimensions, but strongly distorted in others. Moreover, we demonstrate the potential usage of the derived statistics in learning strategies. First, small eigenvalues that induce flatness can be connected to a norm-based capacity measure of generalization ability. Second, the maximum eigenvalue that induces the distortion enables us to quantitatively estimate an appropriately sized learning rate for gradient methods to converge.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Scaling & Learning Theory, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Scaling & Learning Theory

In Statistical Learning Theory

Citations in the corpus

Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, methodology): “F By Cor 2.4, we can then justify the computations of Amari et al. (2018); Karakida et al. (2018) rigorously.”
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10, 000-Layer Vanilla Convolutional Neural Networks (2018, methodology): “…work has used mean ﬁeld theory to build a theoretical understanding of neural networks with random parameters (Poole et al., 2016; Schoenholz et al., 2017; Yang & Schoenholz, 2017; Schoenholz et al., 2017; Karakida et al., 2018; Hayou et al., 2018; Hanin & Rolnick, 2018; Yang & Schoenholz, 2018).”
Neural Tangents: Fast and Easy Infinite Neural Networks in Python (2019, methodology): “initialization (Cho & Saul, 2009; Daniely et al., 2016; Poole et al., 2016; Schoenholz et al., 2016; Yang & Schoenholz, 2017; Xiao et al., 2018; Li & Nguyen, 2019; Pretorius et al., 2018; Hayou et al., 2018; Karakida et al., 2018; Blumenfeld et al., 2019; Hayou et al., 2019).”
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (2018, background): “…theory to explore questions of trainability and expressivity in fully-connected networks and residual networks (Poole et al., 2016; Schoen-holz et al., 2017; Yang & Schoenholz, 2017; Schoenholz et al., 2017; Karakida et al., 2018; Hayou et al., 2018; Hanin & Rolnick, 2018; Yang & Schoenholz, 2018).”
Information Geometry of Orthogonal Initializations and Training (2018, methodology): “Karakida et al. [18] derived an upper bound on the maximum eigenvalue in terms of the sum of the Frobenius norms of its constituent blocks.”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “[29] studies the spectrum of the Fisher information matrix, which share the nonzero eigenvalues with the NTK.”
Disentangling trainability and generalization in deep learning (2019, background): “, 2019), the spectrum of the fisher (Karakida et al., 2018), a range of activation functions (Hayou et al.”
Second-order Optimization Method for Large Mini-batch: Training ResNet-50 on ImageNet in 35 Epochs (2018, background): “The eigenvalue distribution of the Fisher information matrix (FIM) of deep neural networks is known to have an extremely long tail [16], where most of the eigenvalues are close to zero.”
The power of quantum neural networks (2020, methodology): “In this way, we can explicitly relate the effective dimension to model trainability [34].”
Implicit Regularization via Neural Feature Alignment (2020, background): “Recent analytical results for wide random neural networks also point to such a pathological structure of the spectrum (Karakida et al., 2019a,b).”
Freeze and Chaos for DNNs: an NTK view of Batch Normalization, Checkerboard and Boundary Effects (2019, background): “In particular, the Neural Tangent Kernel (NTK) [12] illustrates the effect of architecture on the training of DNNs and also describes their loss surface[13].”
Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks (2018, background): “The eigenvalue distribution of the Fisher information matrix (FIM) of deep neural networks is known to have an extremely long tail [16], where most of the eigenvalues are close to zero.”
The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study (2019, methodology): “It follows that the stability bound for the bare learning rate (cid:15) scales like 1 /w for the standard scheme (Karakida et al., 2017), while it remains constant for the NTK scheme.”
Mean-field Behaviour of Neural Tangent Kernel for Deep Neural Networks (2019, background): “…a function space with respect to the NTK. Du et al. (2019) used a similar approach to prove that full batch GD converges to global minima for shallow neural networks, and Karakida et al. (2018) linked the Fisher information matrix to the NTK, studying its spectral distribution for inﬁnite width NN.”
Generalization of Two-layer Neural Networks: An Asymptotic Viewpoint (2020, methodology): “In addition, similar tools have been used to study two-layer neural networks (Louart et al., 2018; Pennington and Worah, 2017) and related quantities such as the Fisher matrix (Karakida et al., 2018; Pennington and Worah, 2018).”
Information-Theoretic Local Minima Characterization and Regularization (2019, methodology): “In speciﬁc, Karakida et al. (2019) applied mean ﬁeld theory to study the statistics of the spectrum and the appropriate size of the learning rate.”