Large Scale Structure of Neural Network Loss Landscapes

By Stanislav Fort, Stanislaw Jastrzebski. 2019

106 citations recorded by the source corpus.

There are many surprising and perhaps counter-intuitive properties of optimization of deep neural networks. We propose and experimentally verify a unified phenomenological model of the loss landscape that incorporates many of them. High dimensionality plays a key role in our model. Our core idea is to model the loss landscape as a set of high dimensional \emph{wedges} that together form a large-scale, inter-connected structure and towards which optimization is drawn. We first show that hyperparameter choices such as learning rate, network width and $L_2$ regularization, affect the path optimizer takes through the landscape in a similar ways, influencing the large scale curvature of the regions the optimizer explores. Finally, we predict and demonstrate new counter-intuitive properties of the loss-landscape. We show an existence of low loss subspaces connecting a set (not only a pair) of solutions, and verify it experimentally. Finally, we analyze recently popular ensembling techniques for deep networks in the light of our model.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector (2021, neutral): “On the other side, many researches have focused on the connectivity in neural network loss landscapes [12,13].”
What is being transferred in transfer learning? (2020, background): “It has been observed in the literature that any two minimizers of a deep network can be connected via a non-linear low-loss path [11, 9, 10].”
Deep Ensembles: A Loss Landscape Perspective (2019, methodology): “We construct a low-loss tunnel between different optima using the procedure proposed by Fort & Jastrzebski (2019), which is a simpliﬁcation of the procedures proposed in Garipov et al. (2018) and Draxler et al. (2018).”
Questions for Flat-Minima Optimization of Modern Neural Networks (2022, background): “…clear interpretations, as pointed out by Frankle et al. (2020), or (b) yield trivial network connectivity results, such as non-linear low-loss paths, which can be found for any two network minimizers (Draxler et al., 2018; Garipov et al., 2018a; Gotmare et al., 2019; Fort & Jastrzebski, 2019).”
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks (2021, background): “It has been observed in the literature that any two minimizers of a deep network can be connected via a non-linear low-loss path (Garipov et al., 2018; Draxler et al., 2018; Fort & Jastrzebski, 2019).”
Learning Neural Network Subspaces (2021, methodology): “This deﬁnition formalizes that in Fort & Jastrzebski (2019).”
Emergent properties of the local geometry of neural loss landscapes (2019, methodology): “e curvature as measured by the Hessian statistic Trace(H)=jjHjj. This statistic takes larger positive values when typical randomly chosen directions in parameter space exhibit more positive curvature [17, 18]. Thus overall, the ease of optimizing over low dimensional hyperplanes correlates with inte”