An Empirical Analysis of Deep Network Loss Surfaces

By Daniel Jiwoong Im, Michael Tao, Kristin Branson. 2016

48 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

Citations in the corpus

Weight Standardization (2019, background): “To name a few, [15] shows that BN is able to make optimization trajectories more robust to the parameter initialization.”
A Closer Look at Memorization in Deep Networks (2017, background): “, 2013), or the shape of the loss minima found by SGD (Im et al., 2016; Chaudhari et al., 2016; Keskar et al., 2016).”
Sharp Minima Can Generalize For Deep Nets (2017, background): “Im et al. (2016) have plotted how the loss function landscape was affected by batch normalization.”
Visualizing the Loss Landscape of Neural Nets (2017, background): “It was also used in [21] to show that different optimization algorithms find different local minima within the 2D projected space.”
How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift) (2018, background): “[9] observe that networks with BatchNorm tend to have optimization trajectories that rely less on the parameter initialization.”
Visualizing and Understanding the Effectiveness of BERT (2019, background): “In the computer vision community, many efforts have been made to visualize the loss function, and figure out how the geometry of a loss function affects the generalization (Goodfellow and Vinyals, 2015; Im et al., 2016; Li et al., 2018).”
Multi-class Classification without Multi-class Labels (2019, background): “The loss surfaces are plotted with the function (Goodfellow et al., 2014; Im et al., 2016; Li et al., 2017): where θ ∗ are the parameters of the model trained with loss function L and labeled dataset D = ( X, Y ) .”
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (2018, methodology): “This includes: work on large-batch learning and the generalization gap [148, 72, 64, 57, 67, 131, 66, 146, 94, 151, 152]; work on Energy Landscape approaches to NN training [70, 149, 44, 123, 30, 29, 27, 65, 47, 12, 104, 150, 50, 83, 82, 95]; work on using weight matrices or properties of weight mat”
Interpreting Adversarial Robustness: A View from Decision Surface in Input Space (2018, background): “It is commonly believed that a neural network’s generalization is correlated to its geometric properties of the loss surface, i.e. the ﬂatness of the local minima in parameter space (Kawaguchi (2016); Im et al. (2017)).”
Deep Supervised Learning Using Local Errors (2017, background): “, when it includes all the network parameters (Choromanska et al., 2015; Im et al., 2016).”
Exploring loss function topology with cyclical learning rates (2017, methodology): “We adopted the method of Goodfellow et al. (2014) and Im et al. (2016) to compare the solutions obtained at the end of each learning rate cycle (i.e., at iterations 20 , 000 , 40 , 000 , ... ).”
Super-Convergence: Very Fast Training of Residual Networks Using Large Learning Rates (2017, background): “simplest, which is letting the learning rate change linearly (similarly Jastrzębski et al. (2017) suggest discrete jumps).”