The Goldilocks zone: Towards better understanding of neural network loss landscapes

By Stanislav Fort, Adam Scherlis. 2018

60 citations recorded by the source corpus.

We explore the loss landscape of fully-connected and convolutional neural networks using random, low-dimensional hyperplanes and hyperspheres. Evaluating the Hessian, H, of the loss function on these hypersurfaces, we observe 1) an unusual excess of the number of positive eigenvalues of H, and 2) a large value of Tr(H)/||H|| at a well defined range of configuration space radii, corresponding to a thick, hollow, spherical shell we refer to as the Goldilocks zone. We observe this effect for fully-connected neural networks over a range of network widths and depths on MNIST and CIFAR-10 datasets with the ReLU and tanh non-linearities, and a similar effect for convolutional networks. Using our observations, we demonstrate a close connection between the Goldilocks zone, measures of local convexity/prevalence of positive curvature, and the suitability of a network initialization. We show that the high and stable accuracy reached when optimizing on random, low-dimensional hypersurfaces is directly related to the overlap between the hypersurface and the Goldilocks zone, and as a corollary demonstrate that the notion of intrinsic dimension is initialization-dependent. We note that common initialization techniques initialize neural networks in this particular region of unusually high convexity/prevalence of positive curvature, and offer a geometric intuition for their success. Furthermore, we demonstrate that initializing a neural network at a number of points and selecting for high measures of local convexity such as Tr(H)/||H||, number of positive eigenvalues of H, or low initial loss, leads to statistically significantly faster training on MNIST. Based on our observations, we hypothesize that the Goldilocks zone contains an unusually high density of suitable initialization configurations.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Deep Ensembles: A Loss Landscape Perspective (2019, background): “This was geometrically understood and extended in (Fort & Scherlis, 2019).”
Stiffness: A New Perspective on Generalization in Neural Networks (2019, background): “Fort & Scherlis (2018) extends the analysis in Li et al. (2018) to demonstrate the role of initialization on the effective dimensionality.”
Large Scale Structure of Neural Network Loss Landscapes (2019, methodology): “[Fort and Scherlis, 2018] This suggests the special role of the radial direction as well as the distributed nature of the low-loss manifold.”
The Break-Even Point on Optimization Trajectories of Deep Neural Networks (2020, methodology): “Recent work has shown that the largest eigenvalues of H can grow quickly in the early phase of training (Keskar et al., 2017; Sagun et al., 2017; Fort & Scherlis, 2019; Jastrzebski et al., 2018).”
Emergent properties of the local geometry of neural loss landscapes (2019, methodology): “This statistic takes larger positive values when typical randomly chosen directions in parameter space exhibit more positive curvature [17, 18].”
Understanding the impact of entropy on policy optimization (2018, background): “Firstly, given that we are optimizing probability distributions, many reparameterizations can result in the same distribution.”