Geometry of Neural Network Loss Surfaces via Random Matrix Theory

By Jeffrey Pennington, Yasaman Bahri. 2017

169 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

Citations in the corpus

Neural Tangent Kernel: Convergence and Generalization in Neural Networks (2018, background): “A number of results [3, 13, 14] suggest that for wide enough networks, there are very few “bad” local minima, i.”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, methodology): “[50] A similar decomposition has been used in [25, 77, 78].”
On the Convergence Rate of Training Recurrent Neural Networks (2018, methodology): “There is huge literature on using the mean-ﬁeld theory to study neural networks [10, 14, 25, 30, 34, 38–40, 42, 47, 56–58].”
Estimating the Spectral Density of Large Implicit Matrices (2018, methodology): “A simple approximation which captures this additive structure of the Hessian is outlined in Pennington and Bahri (2017), and involves approximating H l as a Wishart matrix and H f as a Wigner matrix.”
Lightlike Neuromanifolds, Occam's Razor and Deep Learning (2019, background): “It is reasonable [30] to assume the parameter-output Jacobian matrix ∂y/∂ Θ follows an i.i.d. Gaussian distribution.”
Deep Learning Through the Lens of Example Difficulty (2021, background): “Examples of such work include studies of eigenvalues of the Hessian and the geometry of the loss landscape (Ghorbani et al., 2019; Yao et al., 2020; Sagun et al., 2016; Li et al., 2018; Pennington and Bahri, 2017; Sagun et al., 2018), studies of margin and effective generalization measures (Long and”
Cost function dependent barren plateaus in shallow parametrized quantum circuits (2021, background): “While scaling results have been obtained for classical neural networks 45 , very few such results exist for the trainability of parametrized quantum circuits, and more generally for quantum neural networks.”
The Emergence of Spectral Universality in Deep Networks (2018, background): “See [7] for a pedagogical introduction, and [3, 8] for prior work applying free probability to deep learning.”
Asymmetric Valleys: Beyond Sharp and Flat Local Minima (2019, background): “Neural network landscape analysis is an active and exciting area [16, 34, 15, 40, 49, 10, 43].”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “[41] Jeffrey Pennington and Yasaman Bahri.”
Universal statistics of Fisher information in deep neural networks: mean field approach (2018, methodology): “butions. This Gaussian assumption is popular in the following theories of neural networks: typical evaluation approach [2, 4], random initialization of training [5, 11], the random matrix formulation [17, 18], and the student-teacher formulation [12, 19]. 2.2 Fisher information matrix (FIM) We focus”
PyHessian: Neural Networks Through the Lens of the Hessian (2019, methodology): “Hessian eigenvalues of small NN models were analyzed [22, 23]; and the work of [21] studied the geometry of NN loss landscapes by computing the distribution of Hessian eigenvalues at critical points.”
On the distance between two neural networks and the stability of learning (2020, background): “Pennington and Bahri [31] do study curvature, but rely on random matrix models to make progress.”
The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size. (2018, background): “This relates to Figure 1 in [Pennington and Bahri, 2017], where under RMT assumptions the authors investigated the interplay between these three quantities.”
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (2018, background): “different Heavy-Tailed Universality classes [46, 32, 18, 25, 20, 5, 111, 7, 38, 17, 90, 8, 99]; other work on RMT approaches [133, 120, 114, 112, 87, 87, 137, 85, 126]; other work on statistical physics approaches [132, 52, 117, 125, 137, 119, 113]; work on fitting to noisy versus reliable signal [1”
Asymptotics of Wide Networks from Feynman Diagrams (2019, background): “For additional works on wide networks, including relating them to Gaussian processes, see [9–22].”
Towards Understanding Regularization in Batch Normalization (2018, background): “Many studies analysed neural networks (Opper et al., 1990; Saad & Solla, 1996; Bs & Opper, 1998; Pennington & Bahri, 2017; Zhang et al., 2017b; Brutzkus & Globerson, 2017; Raghu et al., 2017; Mei et al., 2016; Tian, 2017).”