Visualizing the Loss Landscape of Neural Nets
By Hao Li, Zheng Xu, Gavin Taylor, Tom Goldstein. 2017
2,377 citations recorded by the source corpus.
Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well-known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effects on the underlying loss landscape, are not well understood. In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple "filter normalization" method that helps us visualize loss function curvature and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture affects the loss landscape, and how training parameters affect the shape of minimizers.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Jianfeng Gao
- Mohit Bansal
- Quoc V. Le
- Percy Liang
- Sergey Levine
- Dacheng Tao
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Bernt Schiele
- Jitendra Malik
- Alan L. Yuille
In Optimization & Training
- Martin Jaggi
- Simon Lacoste-Julien
- Sebastian U. Stich
- Peter Richtárik
- Frank Hutter
- Quoc V. Le
- Yoshua Bengio
- Michael I. Jordan
In Scaling & Learning Theory
- Andrew Gordon Wilson
- Michael W. Mahoney
- Stefano Soatto
- Charles H. Martin
- Pavel Izmailov
- Samet Oymak
- Alessandro Achille
- Hanie Sedghi
Citations in the corpus
- An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (2019, background): “Finally, Li et al. [19] recently hypothesized that adding residual connections significantly smooths the optimization landscape, producing a series of compelling two-dimensional visualizations.”
- A Closer Look at the Optimization Landscapes of Generative Adversarial Networks (2019, methodology): “Another extension was proposed by (Li et al., 2018a) where they use contour plots to look at the 2D loss surface defined by two directions chosen appropriately.”
- Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian (2019, background): “Also see [36, 30] for interesting visualization of the optimization and generalization landscape.”
- Learning to Learn with Generative Models of Neural Network Checkpoints (2022, methodology): “We visualize the test error landscape for an MNIST MLP via parameter space PCA directions (Li et al., 2018).”
- How To Backdoor Federated Learning (2018, background): “This produces poor models in the case of non-convex loss functions [38] and/or if the training data comes from a diverse set of users [25].”
- Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “To give an example of how visualization can be helpful in troubleshooting models generally, Li et al. (2017) visualized the loss landscape of neural networks and, as a result, found answers to questions about the selection of the batch size, optimizer, and network architecture.”
- Adversarial Training for Free! (2019, methodology): “3 we plot the cross-entropy loss projected along two directions in image space for the first few validation examples of CIFAR-10 [Li et al., 2018].”
- Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, methodology): “It has been observed that the optimization algorithm being used significantly impacts the properties of the optima reached (Im et al., 2016; Li et al., 2018).”
- Wireless Network Intelligence at the Edge (2018, methodology): “On this account, a filter normalization technique [99] has been proposed, which heuristically guarantees a tight relationship between the resulting landscape and the final output.”
- Don't Use Large Mini-Batches, Use Local SGD (2018, methodology): “(2014) and then widely used to study the “sharpness” and “flatness” of different minima (Keskar et al., 2016; Dinh et al., 2017; Li et al., 2018).”
- Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks (2020, methodology): “Thus, we employ filter normalization to ensure that our calculation is invariant to scaling (Li et al., 2018).”
- A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, positive): “This result harmonizes with other work analyzing the spectra of the Hessian for DNN training [37, 57].”
- How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift) (2018, background): “To understand why, recall that in a vanilla (non-BatchNorm), deep neural network, the loss function is not only non-convex but also tends to have a large number of “kinks”, flat regions, and sharp minima [17].”
- Critical Learning Periods in Deep Neural Networks (2017, methodology): “It is also worth noticing how there is evidence that convergence to flat minima (minima with low curvature) in a DNN correlates with a good generalization performance (Hochreiter & Schmidhuber, 1997; Li et al., 2018; Chaudhari et al., 2017; Keskar et al., 2017).”
- Representational Continuity for Unsupervised Continual Learning (2021, methodology): “We use the visualization tool from Li et al. (2018) that searches the task loss surface by repeatedly adding random perturbations to model weights.”