An empirical analysis of the optimization of deep network loss surfaces

By Daniel Jiwoong Im, Michael Tao, Kristin Branson. 2016

66 citations recorded by the source corpus.

The success of deep neural networks hinges on our ability to accurately and efficiently optimize high-dimensional, non-convex functions. In this paper, we empirically investigate the loss functions of state-of-the-art networks, and how commonly-used stochastic gradient descent variants optimize these loss functions. To do this, we visualize the loss function by projecting them down to low-dimensional spaces chosen based on the convergence points of different optimization algorithms. Our observations suggest that optimization algorithms encounter and choose different descent directions at many saddle points to find different final weights. Based on consistency we observe across re-runs of the same stochastic optimization algorithm, we hypothesize that each optimization algorithm makes characteristic choices at these saddle points.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

A Closer Look at Memorization in Deep Networks (2017, background): “…proposed before, including looking at generalization error, trajectory length evolution (Raghu et al., 2016), analyzing Jacobians associated to different layers (Wang; Saxe et al., 2013), or the shape of the loss minima found by SGD (Im et al., 2016; Chaudhari et al., 2016; Keskar et al., 2016).”
Sharp Minima Can Generalize For Deep Nets (2017, background): “Im et al. (2016) have plotted how the loss function landscape was affected by batch normalization.”
Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, methodology): “It has been observed that the optimization algorithm being used significantly impacts the properties of the optima reached (Im et al., 2016; Li et al., 2018).”
Visualizing the Loss Landscape of Neural Nets (2017, methodology): “Smith & Topin (2017) use the same 1D interpolation technique to show different minima and the “peaks” between them, while Im et al. (2016) plot the line between minima obtained via different optimizers.”
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization (2019, background): “…high-dimensional loss landscapes of overparameterized deep neural networks (Dauphin et al., 2014; Choromanska et al., 2014; Goodfellow et al., 2014; Im et al., 2016; Wu et al., 2017; Liao & Poggio, 2017; Cooper, 2018; Novak et al., 2018), and to the implicit regularization properties of SGD…”
Quantitatively Evaluating GANs With Divergences Proposed for Training (2018, methodology): “The Generative Adversarial Metric (Im et al., 2016a) measures the relative performance of two GANs by measuring the likelihood ratio of the two models.”
Multi-class Classification without Multi-class Labels (2019, background): “The loss surfaces are plotted with the function (Goodfellow et al., 2014; Im et al., 2016; Li et al., 2017): where θ ∗ are the parameters of the model trained with loss function L and labeled dataset D = ( X, Y ) .”
Exploring loss function topology with cyclical learning rates (2017, methodology): “We adopted the method of Goodfellow et al. (2014) and Im et al. (2016) to compare the solutions obtained at the end of each learning rate cycle (i.e., at iterations 20, 000, 40, 000, ...).”