Qualitatively characterizing neural network optimization problems

By Ian J. Goodfellow, Oriol Vinyals. 2014

577 citations recorded by the source corpus.

Training neural networks involves solving large-scale non-convex optimization problems. This task has long been believed to be extremely difficult, with fear of local minima and other obstacles motivating a variety of schemes to improve optimization, such as unsupervised pretraining. However, modern neural networks are able to achieve negligible training error on complex tasks, using only direct training with stochastic gradient descent. We introduce a simple analysis technique to look for evidence that such networks are overcoming local optima. We find that, in fact, on a straight path from initialization to solution, a variety of state of the art neural networks never encounter any significant obstacles.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

An Empirical Model of Large-Batch Training (2018, background): “The loss tends to be fairly smooth when interpolating between the start and end of training [GVS14].”
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (2019, background): “…explanations of the efﬁcacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6], qualitative studies and visualizations of the loss surface [11], and characterization of the intrinsic dimensionality of the loss [18].”
High-dimensional dynamics of generalization error in neural networks (2017, background): “ behavior is a hallmark of deep dynamics initialized close to the saddle point where all weights are zero, and has been found for training error dynamics as well (Saxe, McClelland, and Ganguli, 2014; Goodfellow, Vinyals, and Saxe, 2015). Remarkably, in the reduction the entire eect of depth is compr”
A Closer Look at the Optimization Landscapes of Generative Adversarial Networks (2019, background): “Goodfellow et al. (2015) proposed to look at the linear path between two points in parameter space and show that neural networks behave similarly to a convex loss function along this path.”
A Convergence Theory for Deep Learning via Over-Parameterization (2018, background): “[25] that GD/SGD will not be trapped in local minima.”
Measuring the Intrinsic Dimension of Objective Landscapes (2018, background): “Goodfellow et al. (2015) showed another property: that paths directly from the initial point to the ﬁnal point of optimization are often monotonically decreasing.”
On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization (2018, background): “Other works characterize other aspects of objective landscapes, for example Safran & Shamir (2016) showed that under certain conditions a monotonically descending path from initialization to global optimum exists (in compliance with the empirical observations of Goodfellow et al. (2014)).”
Decentralized Bayesian Learning over Graphs (2019, neutral): “Without modelling the correlation between the weights and bias of the agents across the network, different random initialization at each agent can lead to different local minima and result in diverging local models at the agents [29].”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “(2017) argue that thus mainly the ratio ρ controls the stochastic noise and training dynamics, and that ρ positively correlates with wider minima and better generalization— matching with recent experimental successes of ImageNet training (Goyal et al., 2017). This explanation fails in the large batc”
Distributed learning of deep neural network over multiple agents (2018, background): “Additionally, federated averaging fails to achieve optimal accuracy for higher numbers of clients since general non-convex optimization averaging models in parameter space could produce an arbitrarily bad model (phenomenon described in [39]).”
Communication-Efficient Learning of Deep Networks from Decentralized Data (2016, background): “Recent work indicates that in practice, the loss surfaces of sufficiently over-parameterized NNs are surprisingly wellbehaved and in particular less prone to bad local minima than previously thought [11, 17, 9].”
Federated Learning with Non-IID Data (2018, positive): “Note that the initial weights are also identical across the clients to avoid accuracy loss according to Remark 3.3, which is consistent with the signiﬁcant increase in the loss when averaging models with different initialization [3, 23].”
Visualizing the Loss Landscape of Neural Nets (2017, methodology): “[13] found that loss functions appear to lack local minima along the minimization trajectory.”
Entropy-SGD: biasing gradient descent into wide valleys (2016, background): “All these results suggest that the energy landscape of deep neural networks should be easy to optimize and they more or less hold in practice — it is easy to optimize a prototypical deep network to near-zero loss on the training set (Hardt et al., 2015; Goodfellow & Vinyals, 2015).”
Understanding the Role of Training Regimes in Continual Learning (2020, background): “Works as [11, 23] make similar claims for generic models.”
Visualizing and Understanding the Effectiveness of BERT (2019, methodology): “The loss curve is plotted by linear interpolation between θ0 and θ1 (Goodfellow and Vinyals, 2015).”