Essentially No Barriers in Neural Network Energy Landscape

By Felix Dräxler, Kambis Veschgini, Manfred Salmhofer, Fred A. Hamprecht. 2018

557 citations recorded by the source corpus.

Training neural networks involves finding minima of a high-dimensional non-convex loss function. Knowledge of the structure of this energy landscape is sparse. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, the paths are essentially flat in both the training and test landscapes. This implies that neural networks have enough capacity for structural changes, or that these changes are small between minima. Also, each minimum has at least one vanishing Hessian eigenvalue in addition to those resulting from trivial invariance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Scaling & Learning Theory

Citations in the corpus

An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (2019, background): “…Resnet-32 with no batch normalization, a small ball around 0 accounts for almost 50% of the total L 1 energy of the Hessian eigenvalues for a converged model (the L 1 reﬂects the loss function Thus to achieve successful optimization, we are forced to optimize these slowly converging directions 6 .”
On The Impact of Machine Learning Randomness on Group Fairness (2023, background): “One possible explanation of this immediate impact is the presence of no energy valleys in deep learning loss landscape between minima of separately trained models [18, 24].”
A Closer Look at the Optimization Landscapes of Generative Adversarial Networks (2019, background): “( More complex paths can be considered to assess whether different minima are connected Draxler et al., 2018).”
Wireless Network Intelligence at the Edge (2018, methodology): “A recent study [100] detours this problem by using a minimum energy path (MEP) method in statistical physics.”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, neutral): “[13] Draxler, F.”
Averaging Weights Leads to Wider Optima and Better Generalization (2018, background): “Recent work [Garipov et al., 2018, Draxler et al., 2018] shows that local optima found by SGD can be connected by simple curves of near constant loss.”
Rehearsal revealed: The limits and merits of revisiting samples in continual learning (2021, neutral): “This results in a fixed ellipsoid approximation of the loss contours in the minimum, which is contradicted by our linear interpolation plots in Figure 2 and other work on loss landscapes [11, 28].”
Linear Mode Connectivity in Multitask and Continual Learning (2020, methodology): “Mode connectivity (Draxler et al., 2018; Garipov et al., 2018), is a novel tool to understand the loss landscape of deep neural networks.”
What is being transferred in transfer learning? (2020, background): “It has been observed in the literature that any two minimizers of a deep network can be connected via a non-linear low-loss path [11, 9, 10].”
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs (2018, background): “[5] simultaneously and independently discovered the existence of curves connecting local optima in DNN loss landscapes.”
Rigging the Lottery: Making All Tickets Winners (2019, background): “However, this is only one of inﬁnitely many paths between the two points (Garipov et al., 2018; Draxler et al., 2018) and does not imply the nonex-istence of such a path.”
Deep Ensembles: A Loss Landscape Perspective (2019, methodology): “We construct a low-loss tunnel between different optima using the procedure proposed by Fort & Jastrzebski (2019), which is a simpliﬁcation of the procedures proposed in Garipov et al. (2018) and Draxler et al. (2018).”
On the Spectral Bias of Neural Networks (2018, methodology): “We use the formulation of NEB without springs (Draxler et al., 2018).”
Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, methodology): “We demonstrate that different points along the valleys of high posterior density (low loss) connecting pairs of independently trained optima (Garipov et al., 2018; Draxler et al., 2018; Fort & Jastrzebski, 2019) correspond to different predictive functions.”
Deep learning generalizes because the parameter-function map is biased towards simple functions (2018, background): “(2017); Draxler et al. (2018)) (8)See Appendix E, for definition of the Lempel-Ziv based complexity measure”
Towards Deep Learning Models Resistant to Large Perturbations (2020, background): “Visualization 3 : Mode connectivity (Dr¨axler et al., 2018; Freeman & Bruna, 2016; Garipov et al., 2018) is an unexpected phenomenon in the loss landscape of deep nets.”
Questions for Flat-Minima Optimization of Modern Neural Networks (2022, background): “…clear interpretations, as pointed out by Frankle et al. (2020), or (b) yield trivial network connectivity results, such as non-linear low-loss paths, which can be found for any two network minimizers (Draxler et al., 2018; Garipov et al., 2018a; Gotmare et al., 2019; Fort & Jastrzebski, 2019).”
Learning Neural Network Subspaces (2021, background): “Garipov et al. (2018); Draxler et al. (2018) determine that independently trained models are connected by a curve in weight space along which loss remains low.”