Stuck in a What? Adventures in Weight Space

By Zachary Chase Lipton. 2016

19 citations recorded by the source corpus.

Deep learning researchers commonly suggest that converged models are stuck in local minima. More recently, some researchers observed that under reasonable assumptions, the vast majority of critical points are saddle points, not true minima. Both descriptions suggest that weights converge around a point in weight space, be it a local optima or merely a critical point. However, it’s possible that neither interpretation is accurate. As neural networks are typically over-complete, it’s easy to show the existence of vast continuous regions through weight space with equal loss. In this paper, we build on recent work empirically characterizing the error surfaces of neural networks. We analyze training paths through weight space, presenting evidence that apparent convergence of loss does not correspond to weights arriving at critical points, but instead to large movements through flat regions of weight space. While it’s trivial to show that neural network error surfaces are globally non-convex, we show that error surfaces are also locally nonconvex, even after breaking symmetry with a random initialization and also after partial training.

Subfields: Pharmacy, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Pharmacy

In Optimization & Training

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, background): “ thus raises the question of an appropriate framework to understand generalizations of DNNs. Recent works suggest that the two questions above are closely connected. Numerical and theoretical studies [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17] show that in the overparametrized regime, the loss land”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “Furthermore, studies of the Hessian of the loss function [24–26] and of the learning dynamics [27, 28] support that the landscape is characterized by an abundance of flat directions, even near its bottom, at odds with traditional glassy systems.”
Visualizing the Loss Landscape of Neural Nets (2017, background): “Several authors have observed that random directions fail to capture the variation in optimization trajectories, including [10, 29, 28, 27].”
A jamming transition from under- to over-parametrization affects generalization in deep learning (2018, background): “Furthermore, studies of the Hessian of the loss function [12–14] and of the learning dynamics [15, 16] support that the landscape is characterized by an abundance of flat directions, even near its bottom, at odds with traditional glasses.”
Distributed Bayesian Learning with Stochastic Natural Gradient Expectation Propagation and the Posterior Server (2015, background): “Moreover, learning often gets stuck not at critical points but in large plateaus of the error surface (Lipton, 2016).”
Comparing dynamics: deep neural networks versus glassy systems (2018, background): “And ﬁnally, Lipton (2016) demonstrates how the weights travel large distances through the ﬂat basins by looking at the principle components of the evolution of the weights.”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, background): “dynamics [29, 2] support that the landscape is characterized by an abundance of flat directions, even near its bottom, at odds with traditional glassy systems.”